2

I need to compress a 646.4GB file on a 1TB external HDD. This HDD currently has about 238.6GB free. However, since the straightforward compression methods create a new file, I'll not be able to compress this file, which is a .vhdx file (a Windows system image). I have access to both Windows (10 Tech Preview) and Unix systems (both Mac and Ubuntu, if that helps).

I either need some way to compress these files in place or some way to split this file into several smaller files without leaving the huge 646.4GB as is (ie, some way that deletes data from this file as it is done using it). Thanks!

Ben Sandeen
  • 209
  • 2
  • 4
  • 11

4 Answers4

3

In windows, right click on the file and select Properties, then General Tab, then Advanced Button, tick compress contents to save disk space. This works for individual files or folders.

Moab
  • 58,769
2

Yes, it can be done.

Proof of concept:

Let me get a file to test with:

cp /bin/sh mylargefile

ls -l
-rw-r--r--  1 hennes  users  137208 Jul  1 20:05 my_large_file

Lets chop it into 3 pieces, first piece will be 50k, second will be 50k, third will be 37k. We start at the end with the third part.

 dd if=my_large_file of=part3 bs=1k skip=100
 33+1 records in
 33+1 records out
 34808 bytes transferred in 0.000232 secs (150046592 bytes/sec)

 ~/test$ ls -l
total 180
-rw-r--r--  1 hennes  users  137208 Jul  1 20:05 my_large_file
-rw-r--r--  1 hennes  users   34808 Jul  1 20:09 part3

Ok, so we can copy part of the file. Now lets truncate the original large file to 100000 bytes

truncate  -s 100000 my_large_file
[hennes@dragon] ~/test$ ls -l
total 144
-rw-r--r--  1 hennes  users  100000 Jul  1 20:17 my_large_file
-rw-r--r--  1 hennes  users   34808 Jul  1 20:09 part3

Compress with your favourite program. E.g.

bzip2 -9 part3
[hennes@dragon] ~/test$ ls -l part3.bz2
-rw-r--r--  1 hennes  users  11773 Jul  1 20:09 part3.bz2

Rinse and repeat:

dd if=my_large_file of=part2 bs=1k skip=50
47+1 records in
47+1 records out
48800 bytes transferred in 0.024526 secs (1989735 bytes/sec)

New dd with different numbers. New truccate with different numbers. ...

If you do this without uttermost care you are likely to mess up!

Above examples are already a mix of KB and KiB's. Take extreem care with your numbers.

Also, take a backup. Which means you already have twice the space so this should not be nessecary. Use only if you need to demo something (e.g. as homework or as a proof of concept during a job interview) and at your own risk.

Hennes
  • 65,804
  • 7
  • 115
  • 169
1

BIG WARN:I CAN'T GUARANTEE THIS SCRIPT CODE SAFE!!

On my Debian Stretch I encounter same problem.
I don't know any released tool do this task, so I make a simple shell script for me($1 is the large file):

For compress:

#!/bin/sh -e
#in-place compress single large file
compressor="lz4"
! test -d ./small-files && mkdir ./small-files
while true; do
  size="$(stat -c%s ${1})"
  block="$((1024*1024*1024))"
  if [ "${size}" -gt "${block}" ] ; then
    tail --bytes "${block}" "$1" | ${compressor} > "./small-files/$((${size}-${block}))"
    sync
    truncate -s "$((${size}-${block}))" "${1}"
    sync
  elif [ "${size}" -gt "0" ] ; then
    tail --bytes "${block}" "$1" | ${compressor} > "./small-files/0"
    sync
    truncate -s "0" "${1}"
    sync
  else
    break
  fi
done
echo "success"

For decompress:

#!/bin/sh -e
#in-place decompress single large file
decompressor="lz4cat"
for size in $(ls -1 ./small-files | sort -n) ; do
  truncate -s "${size}" "./${1}"
  sync
  ${decompressor} "./small-files/${size}" >> "./${1}"
  sync
  rm "./small-files/${size}"
  sync
done
echo "success!"

For checksum:

#!/bin/sh -e
#in-place check compressed single large file
origincat () {
  decompressor="lz4cat"
  for size in $(ls -1 ./small-files | sort -n) ; do
    ${decompressor} "./small-files/${size}"
  done
}
origincat | md5sum
Community
  • 1
illiterate
  • 309
0

If the VHD file has windows on it, which I assume it does because you say "Windows System Image".

You might be better off mounting the VHD. Then use a tool like microsoft's imagex to compress the drive into an image file.

imagex /capture /flags "professional" c: n:\images\windows.wim "Win"
cybernard
  • 14,924