The webserver I was working on was running out of space only to find people had been uploading anywhere from 3Mb to 9MB jpeg images. Nobody needs that kind of image size and quality on the wordpress site.
Put together a script to for find and re-encode jpeg images over a certain size, to a lower quality setting.
find . -type f -size +1000k -name “*.jpg” -exec mogrify -quality 50 {} \;
I had some teething pains getting the ESXi server running (I think some hardware incompatibilities) but for now (knock on wood) everything is stable. At one point i was really considering switching over to an Intel server to avoid comparability issues, but that comes at the price of new hardware 🙁
http://lime-technology.com/forum/index.php?topic=22553.0 :Discussion of running UnRAID under ESXi on an AMD platform.
For future expansion, an eight port HBA SATA card would be welcome (to allow for more drives)
http://lime-technology.com/forum/index.php?topic=43026.msg410578#msg410578 :A great post on SATA HBA controllers and their performance
To be able to trully test the performance of the drives in a NAS I need a proper tool that runs on the NAS unit itself. I’ve seen a lot bench-marking done with bonnie++ but installing it in FreeNAS wasn’t straightforward.
First you need to install a jail, then you compile the program from source. FreeNAS is set up quite nicely to do this, so it’s not too much trouble, but figuring out this info took some time.
Some links for future reference:
https://blog.pivotal.io/labs/labs/high-performing-mid-range-nas-server
https://doc.freenas.org/9.3/freenas_jails.html
http://louwrentius.com/74tb-diy-nas-based-on-zfs-on-linux.html
Now that I’m relatively happy with the network transfer speeds (would be nice if I had a 10Gb LAN; that’s cost prohibitive for now) and I think I’m able to nearly fully saturating the network with file transfers.
The next step is to figure out how the network shares should be set up/accessed. I don’t want another malware going rampant on my network and encrypting files again.
First thought was to make all network shares read only, with the exception of the user data share. The media and other long term storage doesn’t really need r/w access. I can manage those from a separate r/w share, but for every other user, they would get only read.
The user data folder is another story. The user stores their sensitive data here, and having it deleted or messed around with would be crappy. Not the end of the world, as I have a Crashplan service that backup all the user data and can be recovered at any time (even deleted files) but it’s a hassle to get to, slow to re-download the data so I am leaving this as a worse case/house burns down scenario. I’d like to have a better option to recover erased/messed up data in my house.
The ZFS feature of snapshots is fantastic. I’ve used it, and it came in handy on a few occasions. Since I’ve consolidated all the storage on one system (UnRAID) and moved away from FreeNAS, i’d like to find a way to get a similar job done in some way.
Playing with ZFS (and FreeNAS) again.
I was wondering which way I should setup the RAID array for maximum throughput (speed), and with a bit of searching came across a fantastic article (found here)
The author does many comparisons of RAID 10, and 5 (raidz1), 6 (raidz2), for up to 24 drives to show how performance scales with # of drives as well as which raid is chosen.
Also since ZFS offers realtime compression, there are comparisons of with compression on (both lz4 and lzjb) and no compression.
Excellent read and great reference. Saves me a bunch of time.
For my 4 x 1TB drives, I’m going with a raidz1 (RAID5) with compression turned on to lz4. I can’t wait to see what kind of throughput I can get out of these disks.
Sadly I will be limited by the 1Gb ethernet connection at around 100MB/s transfer speeds.
I could always bond a couple of 1Gb connections to increase the network capability….
Installed LEAP 42.1 and Windows 10 on an UEFI system (and GTP HD) and was preparing for the bootloader to be taken over by Windows, but that didn’t happen.
Just in case, this writeup on how to point the windows boot to point to the grub boot loader may come in handy.
Just built a new unRaid setup (which is running as a VM indide ESXi 6.0.0)
Currently have 3 x 2TB drives in the array with no parity drive. This yields 6TB of storage.
Writing to a user share (no cache disk) yields some pretty great access speeds. The array is empty at this point, with no data on it yet, and there was no parity drive installed.
The unRaid machine was near by the windows box from where I ran this test. maybe 6ft of ethernet from windows machine to switch, and another 6ft cable to the unraid box
After transferring 4.7TB of data to the array, I did another speed test. This time there is still no parity drive, but I’m running the test from a bit further away at maybe 30-40ft of cable. Still pretty good speeds.
Doing a parity check (since I just installed a parity drive) is giving me speeds of 120MB/sec (at the beginning of the check). This is very likely close to the max speed of the drives.
Parity check is finished, and ran another speed test to see how the read/write speeds to the array have changed. The differences are smaller then I expected.
Once I get a windows VM running on the same ESXi server, I will do another test to see how much of an effect the network has on these read/write speeds, but I don’t think the network is limiting me.
I am quite happy with these numbers as the old unRaid box was putting out in the order of 30-40MB/s (about 1/2 of the new speeds).
UPDATE: (29 Dec 2015)
Have been using the unraid NAS for over 2 weeks and I just did another speed test. The server has been in good use since and it’s running at round 70% capacity right now.
The speeds are better then before. I added a couple of 32 GB SSD drives I had laying around as cache drives (in raid1 for redundancy). They should take care of slow write speeds.
The more I use UnRaid, and the more I read about it the more I like it.
I do however miss the snapshotting feature of ZFS. I will have to figure out a way to get similar functionality as it’s nice to have access to changed or deleted files after the fact.
Some links of useful info I found when flashing my M1015 to 9211.
I don’t need a RAID (Integrated RAID) aka IR functionality. I want the controller to just act as a dumb controller (Initiator Target), aka IT mode so need to track down the IT version of the firmware.
http://linustechtips.com/main/topic/104425-flashing-an-lsi-9211-8i-raid-card-to-it-mode-for-zfssoftware-raid-tutorial/
https://lime-technology.com/forum/index.php?topic=12767.0
http://lime-technology.com/forum/index.php?topic=26774.0
http://lime-technology.com/forum/index.php?topic=33651.0
I upgraded from PHP5.4. (end of life) to PHP5.6, and lost the functionality of pthreads on my CentOS6.7 VPS.
After much searching and trials, it turns out that I need specify the specific version of pthreads to be installed, otherwise the newest version gets installed. In this case the newest version pthreads-3.0.9 is only available for php7.0
The last version that worked with PHP5.x is pthreads-2.0.10. None of the 3.0.x versions of phteads work with the php 5.x family.
So to install pthreads to work with php 5.6, run:
#pecl install pthreads-2.0.10
Just for reference, if you try to install the latest pthreads (currently 3.0.9) while running php 5.x, you get the message:
pecl/pthreads requires PHP (version >= 7.0.0RC2), installed version is 5.6.15
No valid packages found
install failed
After pthreads are installed, restart the apache web server
#service httpd restart
I had this happen a couple of times so I should document it for future reference.
Copying a laravel project to another account renders some pages (pages that use templates) as blank. The change is to chmod the storage/app, storage/framework and storage/logs folders to 777.
I don’t know if all the folders need that permission, but I didn’t investigate what the minimum permissions required are.