Compare files on FTP and output difference

I would like to setup a basic system on my server where a few users have their own FTP accounts. Each user will be able to upload a simple file like a text or CSV file with one record per line. I would like the system to then compare the contents of that file to a database for duplicates, and then create a new file with the contents of the database + the contents of the new file, without the duplicates which can then be downloaded by the user. It would be nice if on a per user basis we could offer the new file to be downloaded either in full, or as a 'partial' download if the user has already downloaded it once before (so just the new records would be downloaded).