| View unanswered posts | View active topics | It is currently Sun Apr 12, 2020 03:47
|
no space left problem when writing to USB NTFS driveModerators: d242, szaka
Page 1 of 2
| [ 28 posts ] | Go to page1, 2Next |
| Previous topic | Next topic |
no space left problem when writing to USB NTFS drive | Author | Message |
|---|
no space left problem when writing to USB NTFS drive Hi,
I'm having problems when trying to write files into an 1 GB USB 2.0 pendrive. This unit has 968 MB of free space (5 MB are not used from the total 973 MB) and i'm trying to copy 929 MB distributed in 146 files and 8 directories. I'm not getting any error trace in /var/log/messages, /var/log/messages.log, /var/log/syslog or /var/log/daemon.log. I only get traces of mounting and umounting operations in /var/log/syslog and /var/log/daemon.log.
The problem is that the last 8 files can't be copied because the system tells that there is no free remaining space in the USB drive. Those files are touched with 0 KB in the pendrive but it has still 71 MB of free space remaining. The files that can't be copied use 32 MB of disk space.
I first believed that maybe the problem was the internal fragmentation of the disk (i don't know if this is really possible in NTFS though). Then i have noticed that when i get the error, if i umount and mount again the USB drive then, i can then copy the rest of the files without any problem.
What could the problem be?. I don't really now if this is a problem of NTFS-3G or the Linux kernel. To test this i have created a linux bash script that copies some video files and ran it through different configurations.
The system where i have tested are 32-bit GNU/Linux systems on top of Intel(R) Pentium(R) Dual CPU E2200 @ 2.20GHz:
1) Ubuntu 8.10 Intrepid Ibex (Kernel 2.6.27-11-generic #1 SMP, FUSE 2.7.3, NTFS-3G 1.2506)
2) Debian 5.0 Lenny (Kernel 2.6.26-1-686 #1 SMP, FUSE 2.7.4, NTFS-3G 1.2531)
3) Debian 5.0 Lenny (Kernel 2.6.24.4 #1 SMP, FUSE 2.7.4, NTFS-3G 2009.2.1)
The problem is reproduced always and with the same results in those systems configurations.
I have included traces from /var/log/syslog, and /var/log/daemon.log. No traces were written when copying the files or mounting the units in files /var/log/messages and /var/log/messages.log. I have also included the script i have created and the traces of running it in 2) configuration (tree command has been used to show the file sizes before and after the copying process).
Thank you very much for the support and congratulations for NTFS-3G.
File comment: Traces of Debian 5.0 Lenny (Kernel 2.6.26-1-686 #1 SMP, FUSE 2.7.4, NTFS-3G 1.2531) configuration.
traces.tar.gz [4.68 KiB]
Downloaded 841 times
| | Tue Mar 24, 2009 16:16 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive Hi, Please download http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz then run ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2
and send the DEVICE-meta.img.bz2 file to szaka@ntfs-3g.org. Thanks, Szaka
| | Tue Mar 24, 2009 16:38 | Re: no space left problem when writing to USB NTFS drive Hi, Please download http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz then run ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2
and send the DEVICE-meta.img.bz2 file to szaka@ntfs-3g.org. Thanks, Szaka OK. Sent.
| | Tue Mar 24, 2009 16:57 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive I did the following after mounting the two image files under the 'after' and 'before' directories: cp -r after/VideoServer before
dd if=/dev/zero of=before/foo bs=1M
The copy was successful and 'dd' copied a 71 MB file until the volume became full. In short, I could not reproduce your problem. The hardware is not relevant. One difference is that you always used the external FUSE library. NTFS-3G uses the internal one by default (it has all the officially-not-released-yet FUSE fixes, simplification, quality testing). However I'm not aware of any potentially related FUSE problem or what it could case what you experience.
| | Tue Mar 24, 2009 18:53 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive Sorry. I was using the latest development NTFS-3G version for testing. I could reproduce the problem with the latest stable release 2009.2.1. It seems we fixed your problem even before you reported it and we knew about it ;-)
I'll check what's going on and will update.
| | Tue Mar 24, 2009 19:03 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive Hi,
The problem was not easy to reproduce. It seems you just had the right amount of right size files on a right size NTFS volume. The development version had the same problem. It is fixed now at
http://mercurial.creo.hu/repos/ntfs-3g- ... f08d2ef030
It will be included in next stable version which is expected to be released this month. Thanks again.
Best regards, Szaka
| | Wed Mar 25, 2009 02:17 | Re: no space left problem when writing to USB NTFS drive Hi,
I will apply the patch myself meanwhile the next stable version is released.
Thank you very much again for the support and congratulations for the fantastic work.
Antonio.
| | Wed Mar 25, 2009 09:01 | Re: no space left problem when writing to USB NTFS drive Hi again,
I have modified the latest stable release 2009.2.1 with the patch you have send me. Now the problem is that the last files aren't copied because the device has been umounted automatically.
For example, if i mount the device in '/mnt/test_device' and create the directories 'VideoServer/2009/Mar/20/04' and try to copy the files inside it, when reaching the last files (the ones that couldn't be copied before the patch), the device gets umounted and the copy command fails (because the VideoServer directory doesn't exists because the pendrive has been umounted).
Maybe the patch only works in your development version of the driver and i need to wait for the next stable version.
Thanks for the quick support.
| | Wed Mar 25, 2009 10:42 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive Hi,
NTFS-3G and nothing else should unmount the volume. I can only image driver crash. Do you have have any sign of this in your log files?
You could enable core file generation by 'ulimit -c 1000000000' then check for the core file in the root directory. If it happens then could you please check with gdb where it crashes: gdb path-to-ntfs-3g-binary /core then type bt at the gdb prompt?
I can't reproduce the crash problem. It's also possible the patch was not correct though I can't see where it could have gone wrong.
Regards, Szaka
| | Wed Mar 25, 2009 13:14 | Re: no space left problem when writing to USB NTFS drive Hi again,
I haven't been able to reproduce the problem again. I have correctly applied the patch and the first problem i had has never occurred again. Maybe my first test was wrong and i umounted the unit myself by mistake.
I will continue testing and report if i notice something again.
Thanks very much for the help.
| | Wed Mar 25, 2009 15:42 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive Hi again, Can you find anything in your log files related to the unexpected unmount/crash? If the device was unmounted then there must be an ntfs-3g[PID_NUMBER]: Unmounting DEVICE ()
entry in your log file. If there was a crash then the kernel should have also log it in /var/log/messages (messages.log, daemnon.log, etc depending on the distribution). Thanks, Szaka
| | Wed Mar 25, 2009 15:52 | Re: no space left problem when writing to USB NTFS drive Hi again, Can you find anything in your log files related to the unexpected unmount/crash? If the device was unmounted then there must be an ntfs-3g[PID_NUMBER]: Unmounting DEVICE ()
entry in your log file. If there was a crash then the kernel should have also log it in /var/log/messages (messages.log, daemnon.log, etc depending on the distribution). Thanks, Szaka I haven't found anything related to a crash or an error in my log files. No unexpected umount line have been logged. In the following tests i have completed everything worked fine. I will continue testing and report if found something. Regards, Antonio.
| | Wed Mar 25, 2009 15:56 |
Joined: Tue Nov 21, 2006 23:15
Posts: 1648
| Re: no space left problem when writing to USB NTFS drive I can't produce any crash. Some words about the Unmount log entry. Every mount is logged like ntfs-3g [PID] : Mounted /dev/hda3
All of them are coupled with an Unmount log entry as above during normal operation (no power loss, etc). If one Unmount log entry is missing then it means the driver crashed. If you know the time when you did the copy and you can find the relevant Unmount entry in the log file then the volume was unmounted on demand/request (accident or something triggered it for some reason) and we don't need to investigate and try to reproduce crash. The driver never unmounts the volume itself. Unmount always happens following an external event , for instance user device eject request, hotplug, etc. Thanks, Szaka
| | Wed Mar 25, 2009 16:19 | Re: no space left problem when writing to USB NTFS drive please allow me jumping in on this topic:
After having problems using extended (german) characters on the USB-attached NTFS drive mounted with Ubuntu's on-board ntfs, I found your ntfs-3g product last week.
So far, so good, the character handling does work as promised.
But there occurs another problem: this USB drive is also shared as samba resource for some windows computer. And here it comes: trying to copy files via filemanager or syncback to it with 'no space left' - but I can create new files or directories using filemanager ...
And of course I can copy or rsync these same files to the USB disk from an internal ext3 drive (which I shared via samba to get the windows files onto).
The system is running Ubuntu 8.04 LTS (2.6.28-11-server #42-Ubuntu SMP x86_64)
ntfs-3g is 2010-03-06 (so including the afore mentioned patch?)
The mount comes implicitly from /etc/fstab:
/dev/sdd1 /media/usbdisk ntfs-3g default,user,sync,exec 0 0
the drive is a 500GB with only less than 5% free space left
appreciating any help
Michael
| | Wed Mar 31, 2010 10:03 |
Joined: Tue Sep 04, 2007 17:22
Posts: 1286
| Re: no space left problem when writing to USB NTFS drive Hi, And here it comes: trying to copy files via filemanager or syncback to it with 'no space left' - but I can create new files or directories using filemanager ... Is the error showing on a Windows client screen ? ntfs-3g is 2010-03-06 (so including the afore mentioned patch?) Which patch are you mentioning (ntfs-3g-2010.3.6 is the latest stable version) ? /dev/sdd1 /media/usbdisk ntfs-3g default,user,sync,exec 0 0 That is correct, though sync is unusual. the drive is a 500GB with only less than 5% free space left Is 5% what 'df /media/usbdisk' returns ? If the error is shown on a Windows screen, this is caused by Windows checking the space left before copying. For instance if you have 8MB free space and you want to copy a 10MB file into a directory marked for compression, Windows will complains, even though 5MB may be actually needed. What is the free space shown from the Windows point of view ? Regards Jean-Pierre
| | Wed Mar 31, 2010 12:30 | Re: no space left problem when writing to USB NTFS drive Salut, Is the error showing on a Windows client screen ?
Which patch are you mentioning (ntfs-3g-2010.3.6 is the latest stable version) ?
Is 5% what 'df /media/usbdisk' returns ?
What is the free space shown from the Windows point of view ?
yes, this comes on the windows screen, as well in file manager as in CMD-box. within this topic, about one year ago szaka did mention a patch for the then actual version. yes, the 5% are returned by df - and on a 500GB disk, this means still 25GB, which should be really enough for copying even 50 files of 240kB ... and I ran into the problem on wanting to copy 5 of them ... Windows file manager tells me, after subst'ing the UNC, of 17.9GB free of 465GB total. I can copy boot.ini from the c: directory (less than 1 kB), and I can also create new files and directories, but a randomly chosen .inf file of about 5 kB is refused ... The 'sync' parameter I chose to mostly guarantee the coherency of the data, because I must be prepared that the USB disk is unplugged/changed without previous umount'ing - the user does act as on his Windows PC --- Seems I found something: Until now, I tried to copy into some specific -very big- directory on the USB disk; just (by error) copied into some other dir, and it was accepted ... Are there maybe some constraints regarding file and/or subdirectory numbers? thanks in advance Michael
| | Wed Mar 31, 2010 20:40 | Re: no space left problem when writing to USB NTFS drive Are there maybe some constraints regarding file and/or subdirectory numbers?
PS: the directory in question contains 80.861 files in 3.288 directories, totalling to nearly 200 GBytes
| | Wed Mar 31, 2010 20:46 |
Joined: Tue Sep 04, 2007 17:22
Posts: 1286
| Re: no space left problem when writing to USB NTFS drive Hi, within this topic, about one year ago szaka did mention a patch for the then actual version. Ok. This patch is included in all versions since then. Until now, I tried to copy into some specific -very big- directory on the USB disk;
just (by error) copied into some other dir, and it was accepted ... Access times are higher on a near-full disk and on big directories. This could cause some time-out trigger in Samba. PS: the directory in question contains 80.861 files in 3.288 directories, totalling to nearly 200 GBytes ntfs-3g has been successfully tested with much more files in a single directory, but Samba can have some limitations on its own. There are some limits in ntfs (for example no more than 30 directory levels), but they are the same for local access and remote access through Samba. What is the history of this drive ? In what circumstances did the error occur the first time ? What is the exact number of files and directories at a single level (not counting files in subdirectories) ? Can you plug the drive into Windows and start a chksdsk ? Regards Jean-Pierre
| | Fri Apr 02, 2010 08:51 | Re: no space left problem when writing to USB NTFS drive Salut, Ok. This patch is included in all versions since then.
that's what I thought, on first reading the topic i did not regard the dates ... Access times are higher on a near-full disk and on big directories. This could cause some time-out trigger in Samba.
ntfs-3g has been successfully tested with much more files in a single directory, but Samba can have some limitations on its own. There are some limits in ntfs (for example no more than 30 directory levels), but they are the same for local access and remote access through Samba.
Maybe - but this same disk never brought up these problems for over a year, using the 'standard' Ubuntu ntfs driver. The samba system was not touched. Then last week, I came on another, similarly configured machine into the problem with the extended characters, and switched them both into the new ntfs-3g driver, just to keep them both as similar as possible ... What is the history of this drive ? In what circumstances did the error occur the first time ?
What is the exact number of files and directories at a single level (not counting files in subdirectories) ?
The message comes, when trying to copy new files onto the disk into a destination anywhere within the largest top-level directory structure. It does not occur, when I manually create new files or directories within the same directory via Samba, or when I copy very small files to there, or not even when I copy the files into other locations on the drive. It depends really on the file size: I just generated an empty txt file, randomly entered a few lines of characters and saved this file. This small file of -afterwards counted 496 Bytes- I can copy as often as I want. But when I try to append 5 more characters, the saving is deferred as 'not enough space'. So this really seems to be caused by the number of clusters/blocks used on the disk ... These not-syncback'ed files I can Linux-cp into this same location. So it really comes back into a Samba topic (which I agree to not be your problem) ... that came to light after changing the ntfs driver (about which I'd like to get helpful hints from you). For better explanation of the situation: The drive is used as backup media. It is attached to the mentioned Ubuntu machine, which carries a Windows VM. This virtualized Windows was converted from a previously stand-alone file server, that died due to hardware failure (much simpler than to reinstall everything from scratch ...). It does its backup in two stages using 2BrightSparks's SyncBack: 1st level onto an internal ext3 drive every 2 hours, 2nd level once a day to that USB drive. But as I found, the 'disk full' message comes up also on cmd's copy or xcopy ... I want this drive as NTFS to be able to attach it in case of server's failure easily to any user's Windows PC for quick data access. Can you plug the drive into Windows and start a chksdsk ?
That would be possible, but only next week, after the holidays - remote acces is not capable of everything ... Thanks for reading Michael
| | Fri Apr 02, 2010 11:02 |
Joined: Tue Sep 04, 2007 17:22
Posts: 1286
| Re: no space left problem when writing to USB NTFS drive Hi, Maybe - but this same disk never brought up these problems for over a year, using the 'standard' Ubuntu ntfs driver. Do you know what ntfs-3g version was this previous one ? The message comes, when trying to copy new files onto the disk into a destination anywhere within the largest top-level directory structure. Can you remount your device with debug option, trigger an error (try to be minimal), and post the debug information (please gzip it and attach) ? # as root :
umount /media/usbdisk
ntfs-3g -o debug,sync /dev/sdd1 /media/usbdisk 2> debug.log
# trigger an error condition
umount /media/usbdisk
gzip debug.log Regards Jean-Pierre
| | Fri Apr 02, 2010 12:27 | Re: no space left problem when writing to USB NTFS drive Salut again,
thanks for the quick reply.
I did as advised - here#s the debug.log.
Although it was not so easy to umount afterwards - the disk stays busy after accessing it vie file manager from windows.
Had to restart Samba to get it free ...
thanks
Michael
debug.log.gz [45.03 KiB]
Downloaded 656 times
| | Fri Apr 02, 2010 12:53 | Re: no space left problem when writing to USB NTFS drive oh, sorry, forgot something:
the previously used ntfs was the standard version fom Ubuntu8.04, should've been 1.22something ...
| | Fri Apr 02, 2010 12:54 |
Joined: Tue Sep 04, 2007 17:22
Posts: 1286
| Re: no space left problem when writing to USB NTFS drive Hi, I suppose the file you were trying to create was '/server-d/DATEN/Saarxxxxxxxx.DOC', which you were feeding from a file with same name within another directory. I see the first write() was rejected with error code 'Operation not supported'. I suspect you have reached some internal limit which I have to dig more to explain. Can you confirm this is the right file, before I dig ? Also, has something been written to the system log (generally in /var/log/*) during this debug mounting ? *edit* Also : how many files in server-d/DATEN ? ls -ldi <mountpoint>/server-d/DATEN
ls -l <mountpoint>/server-d/DATEN | wc Regards Jean-Pierre
| | Fri Apr 02, 2010 14:23 | Re: no space left problem when writing to USB NTFS drive hey, that's quick ...
yes, that is the file
ls -ldi:
46204 drwxrwxrws 1 root root 65536 2010-04-02 12:47 /media/usbdisk/server-d/DATEN
ls -l ... | wc:
356 3090 22513
I really appreciate your help
Michael
| | Fri Apr 02, 2010 19:16 |
Joined: Tue Sep 04, 2007 17:22
Posts: 1286
| Re: no space left problem when writing to USB NTFS drive Hi Michael, I have examined all the situations in which ntfs-3g returns 'not supported', and there is a single one compatible with your debug data : this is when you try to overwrite an existing compressed file (only appending and clearing are implemented). When copying a file through Samba, you possibly do the following (not sure about it, but this is compatible with the data) : - create the file - truncate the file to target size - write from the beginning In that sequence, the write is an overwrite, which is not implemented if the file is compressed. This explains why you can copy locally, and why you can copy through Samba into another directory. It also explains why you can write small files : compression only takes place when the data does not fit into the inode. To confirm this, please display and post the attribs of the parent directory : getfattr -e hex -n system.ntfs_attrib /media/usbdisk/server-d/DATEN The obvious solution is to remove the compression flag : setfattr -v 0x10000000 -n system.ntfs_attrib /media/usbdisk/server-d/DATEN Now, your disk is nearly full, and compression was probably intended to save space, so you might also try other ways for copying, so that the file is not forced to target size from the beginning (eg something equivalent to 'cat sourcefile > targetfile'). You may also copy to a temporary directory, then copy locally - not move ! - into the compressed directory. Regards Jean-Pierre
| | Sat Apr 03, 2010 08:40 |
Page 1 of 2
| [ 28 posts ] | Go to page1, 2Next |
Who is online | Users browsing this forum: No registered users and 10 guests |
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum
|
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin. |
Comments are closed.