Develop a crawler/parser for wikipedia according to rules

SUMMARY: -------- I want a parser crawler combinatino that retrives a specific html wiki pages, strips out content, remove not needed section, attributes, tags + text. What I expect as Product Owner/Customer: - JAVA Based cmd-line-application. - You can use a Frameworks whatever you want (like JSoup or SAX) that assist you. - You can use Java 1.8 power like Streams - You can use the API from WIKI, if you want - Preparation is always UTF-8 in order to support languages ... - The crawler must work for different languages, like de, it, en, es, pt, ru - Logging FW like log4j, logback or similar - You can use mvn or gradle THE INPUT: ---------- As input you will get specific tuples of URLs, target-folder and target name [login to view URL] --> folder: de/ filename: [login to view URL] [login to view URL] --> folder: de/ filename: [login to view URL] or [login to view URL] --> folder: en/ filename: [login to view URL] [login to view URL] --> folder: en/ filename: [login to view URL] or [login to view URL]%D0%97%D0%BE%D0%BB%D0%BE%D1%82%D0%BE --> folder: ru/ filename: Золото.html [login to view URL]%D0%97%D0%BE%D0%BB%D0%BE%D1%82%D0%BE --> folder: ru/ filename: Самарий.html the method signature should look like this: boolean extractRelevantContentFromUrl(URL, folder, file); Where the return value shows the state if the processing/extraction is successfull. The extraction should take place in folder in the file, eg: extractRelevantContentFromUrl("[login to view URL]", "de", "[login to view URL]"); The resulting String should be checked and the process of generation should be stopped with if a not allowed tag is found (see below). This is especially important after the (my) processes are established. And when wikipedia changes their dom. The resulting HTML must look like this, while preserving the original tag and the original order of appearance: <h1>Header of Page</h1> <p>Text element n</p> <p>Text element n+1</p> <h2>header 2</h2> <p>Text element n</p> <p>Text element n+1</p> <h3>header 3</h3> <p>Text element n</p> <p>Text element n+1</p> <h4>header 4</h4> EXAMPLES: --------- examples are in the attachment. etc SOME RULES: ----------- What I DONT want in the resulting html-string, these are [login to view URL] criterias: - Tables (Completly Removed) - Anchors (The anchor text is still needed..) (<a href="[login to view URL]">Text</a> will be just "Text" without double quotes) - sup (Citation from wiki - completly removed, other sups and subs must stay, like in the example) - Images: Paths has to be rewritten to allow hot linking if a relative path is set (i did not see relative paths for images) - Section: Bibliography, you will need to find a way to omit these information in all languages - Section: External Links, you will need to find a way, that is flexible for all languages - Empty Tags must be removed, like <p></p> or: <div id="someid"></div>, take care of possible whitespaces (trim). - mw-editsection 's ... - toc - in general poluted html: I just want clean tags. - The extraction process must work with all language setups What I expect as deliverables: - The complete source code and pom/gradle file if FW/Libraries are used we can share via private repo - All libraries that are used in the pom/gradle dependency, that i can not get via the default repositories (like private ones) - Clean Code - Documentation where neccessary in english language - It is possible, that the method signature boolean extractRelevantContentFromUrl(URL, folder, filename) is not sufficient. If thats the case, we talk (Mail, Phone, Skype, Facetime or whatever) What I deliver to you: - 10 languages containing of - 118 elements each. You will get a Java Class/Enum that holds the tuple of these (URL, folder, filename) that you can just c&p it. - The money :-)