The open comicbook project work-in-progress report


   Hi, thank you all for all the message and feedback after the release of episode 22!

I spent the first week of June here to finish a side project: an open comicbook project for Pepper&Carrot to be sold and printed by the Krita Foundation and with all the benefits going only to the free software (the one I'm using for doing Pepper&Carrot). I finished the beta version of book 1 last night in Scribus and the book is now ready to be proofread, exported and printed. The git repository is here.

Why did it take me a full week? ... Here is my report if you want technical details:


Screenshot: two instances of Scribus 1.5.4dev running side-by-side on my Linux Mint desktop:
Inner pages on right, cover on left.

The Pepper&Carrot comic mainly uses Krita (for artworks, Krita kra files exported to flat JPG at 95% quality) and Inkscape (SVGs for speech bubbles and text, exported as PNG). For the book project, I used Scribus. Scribus is the only free/libre open-source software to provide a multipages CMYK workflow with advanced text and layout tools. Scribus is a very solid project, and I planned to overlay my artworks and my speech bubbles in it. But Scribus also has a lot of traps: many format (*.pdf, *.eps, *.tiff, *.jpg, *.png) do not really react as expected in the first place and you cannot know it until you try it. One must do the mistakes, do trials-and-errors, export after export, to learn how to prepare in the best way the media before being able to export a good and solid CMYK hi-resolution PDF output ready for the printer. If your project has a small amount of images, it's easy to switch formats and encodings for each of them; but if your project needs a renderfarm to fusion hundreds of Krita files with thousands of SVG files to output the correct files, it can quickly become a tedious exercice.


Dual screen screenshot while scripting and re-rendering translations.

So, it was a long and difficult task to find the best recipe and adapt all the other parts of Pepper&Carrot to be dynamically linked to the book project. The book project is designed to be fully dynamic with the other parts of Pepper&Carrot: if a user submits a correction on one SVG for the webcomic in the git of an episode, the change will be propagated to the book's files. The book project can also switch translation, to ease the work of many micro and local self-publishing effort.

For the book, I used Scribus 1.5.4dev daily build: it was the only way to work around a Dpi problem in the released version making the UI too big or too little. The software is stable and has good ergonomy. It's really easy to use if you have an experience with this type of software (eg. QuarkXpress/Indesign, I was a prepress layout graphic artist in a past life) and has many features, especially for text, paragraph, advanced layout and color management. One specific issue I met with Scribus was related to the PNG hi-resolution I generate with the renderfarm. This PNG are generated from the SVG sources by Inkscape and then handled by Imagemagick and couldn't be read by Inkscape without getting dark edge artifacts around speech bubbles. All pages were affected, no exception. A solution was found in the way PNGs can be encoded, but this change required me to re-render all 5,800 SVG translations of Pepper&Carrot to get the new PNGs...


Issue I met with PNG transparency: speech bubble is a PNG inside an Image Frame laid over the artwork.
Full, bug report and solution here.


Since the creation of Pepper&Carrot, I never ran a full re-render of all the sources; I just keep a good render besides the source. I only regenerate this render in case of a modification of the source. Langage by langage, with review of the translator after the render, this system was working. But I started to get more and more export problems with newer versions of Inkscape.


A common mix of two rendering SVG issues: buggy border radius (bold effect) applying to only a part of the text.
Episode 6 is translated in 34 langages, and sound effects like this appear 4 times in this episode.
136 file to manually fix. If you can fix a file in 2min, it's 4h30 of non-stop work. It was done for all 22 episodes.

Let's be clear: It's not the fault of Inkscape's developers: I have a very wide variety of SVGs like no other project around. I'm totally out-of-the-scope of many common uses of Inkscape because yes, Pepper&Carrot is now super-complex and has the most challenging base of SVGs you could find on the globe (except maybe for Openclipart and Wikimedia Commons).  I'm using special font effects, special text boxes and rare combinations of filters to workaround specific speech bubble shapes and typography on the sounds effect. I also have in the repository source SVGs saved from Inkscape 0.48 to 0.92, from MacOS, Windows or Linux versions, from many different computers with different local settings, with left-to-right langages, right-to-left, using experimental open fonts. It's not a surprise then if this large SVG database is sensitive to breaks: I'm simply pushing the limits of the Inkscape SVG file-format too far!


Original VS Re-rendered translation, with a combo of many new bugs I met:
Right-to-left support punctuation issue + bold radiusx2 + offset in position + missing sound effect.

I first tried to find a way to re-render all within the same Inkscape version; the latest stable is 0.92.1 and converts every previous version while keeping the look of the original. But I had too many failures even with the help of an Inkscape developer directly on the Pepper&Carrot IRC channel. So I decided to build a script to compare before and after versions; cycling over all the SVGs and with a smart automatic detection process to guess when the render was similar and acceptable. I received precious help from the community to build this type of complex script. The script tried to auto-fix everything, taking around 7 second by SVGs analyzed and about 15 hours to cycle over all the database with all my processors screaming their 100% usage on my desktop computer. But many renderings were too different, writing in my console "Irregular rendering issue found" in red. Out of 5,800 SVGs, I had more than 3,000 bad renderings (script sources available here).


The script used to identify problematic rendering in action.
Console on left, Image viewer in the middle showing a detected issue, and on the right a little GUI to interact.
"Is this OK?  .... huh ... No!"  3,000 files had to be reviewed/fixed this way.

Another computer script was able to quickly display the "before and after" in a quick gif animations at the center of my screen and propose a Yes/No interface for the 3,000 irregular SVG renders detected. The idea was simple: the gif pops up on the screen, blinks the before/after versions with a pop-up: Is this OK? Yes or No. If yes, the file commits itself, if no the software opens Inkscape for manual fixes. After days and nights of a brainwashing repetitive work across hundreds of file comparisons, I could only fix a bit less than 300 irregular SVGs and started to get a massive headache (and also loosing any passion for Pepper&Carrot!). 300 files solved for days of work is a tiny dust if I compare to the 3,000 problematic SVGs detected. My first week of June was already finishing, still not a page of the book was done and I was clearly wasting a lot of time to workaround Scribus being picky with PNG encoded background color in transparency. I was also screwing all the database, my health, and in general... my passion for the project.

It was emergency time and I decided to stop: for a single man, fixing this alone would be equal to a full month brainwashing death. I reverted all the converted files even validated, because I was really affecting the quality. I then decided to take another cheap workaround: I installed on my machine multiple versions of Inkscape and made the renderfarm detect the version of the SVG input and redirect the render to the right version of Inkscape. This makes my renderfarm script now super hard to install because you need this setup of special Inkscape versions and having multiple versions of the same software on GNU/Linux is hard.  I simply now render 0.48 and 0.91 SVGs in Inkscape 0.91 and 0.92 SVGs in 0.92. A simple way but not a elegant or a long term solution. Anyway, I wasn't loosing the original project: doing my Scribus book 1, and I could continue with this patch. I was able to finally re-render all the speechbubbles in all langages to workaround my original PNG problem in Scribus and I could get the good PNGs white-backgrounded transparency. Something only Imagemagick can do with special flags.


The inner pages of the book project in Scribus. Totally in sync with the source of the webcomic

Finally, what a story and difficult time! I had the feeling to meet cascading issue a bit like in the famous scene of "Malcolm in the middle" where the father attempt to "Replace a lightbulb" (Imgur gifv link). I hate when this happen and I feel this type of big things will only attack more and more the project. But let's see the bright side: the book project was an opportunity to review the state of all the SVG database, find many bugs in software and SVGs and it will probably make the future brighter for other artists taking the open-source way.

Thanks again to the developers of Inkscape and Scribus, always giving a lot of their time, giving me detailed information and being around when I have to solve this type of big problems. Now the situation is stable again, maintained but at the price of running multiple Inkscape versions on the renderfarm. This situation will not be possible to keep forever and I already feel the subtle long term pressure of this Damocles's sword. Every SVG translations (especially from episode 01 to episode 12) will need at one point a manual review, file by file, and I'll need help from contributors. I'll publish a little how-to about it in a near future.

Now let's turn the page of this big chapter and let's start the next step with the book project.  On my side, I'll start the production of tutorials and Episode 23, I can't wait to draw Pepper&Carrot instead of debugging Pepper&Carrot !

Thank you if you could read it all, If you have questions, please comment: I'll be around.

11 comments

link   Châu   , - Reply

Merci! Awesome work. Like work flow/pipe line story. Give good idea for other people, help know where problems can exist. I create ePub book use Sigil and want convert become print book use Scribus. Book about OpenEXR file format and only have about 50 Inkscape SVG file from version 0.48, and no use special effect or gradient because no know what SVG feature different ePub readers can display.

Every person can download book here (free, no DRM, Tiếng Việt only)
http://www.lulu.com/shop/nh%E1%BB%B1t-c … 07822.html

link   David Revoy   Author , - Reply

Hello Châu,
Thank you, your ebook to Scribus project is very interesting. That's a cool bridge/convertion to do it this way, and I didn't know it was possible.

My problem with "SVG in Scribus" is you can't link SVG ( external SVG file ) to the Scribus *.sla file ; they can be only embed directly as vector graphics. So, to update them, one has to delete and re-import the vector graphics ( or modify them directly in Scribus ). We discussed on the Scribus IRC channel of a new feature: VectorFrame ; working like ImageFrame, but for vector graphics. Just a frame loading an external file, but keeping the vector possibilities. I need this type of feature here to keep my graphics external and be able to switch the files ( eg. switch the lang of speechbubble content ) ; that's why I tested only raster format on my project to keep the feature to link external Image as frame... SVG is fantastic, and with your SVG hygiene ; I'm sure you'll have no problem to update to any version of Inkscape. Thank you for the preview/link of your work!

link   Philip Halabi   , - Reply

Will the book be available in different languages? I would like to buy it in French, German and English.

link   David Revoy   Author , - Reply

Hi Philip,
The Scribus source to do the book are designed to switch the lang, but with the Krita foundation we plan to print only the English version because it's cheaper to print a big amount ( eg. 500 ) of the same file ; and Glénat cover already the french version ; Popcom the German version.

link   igor giuseppe   , - Reply

" All pages were affected, no exception. A solution was found in the way PNGs can be encoded, but this change required me to re-render all 5,800 SVG translations of Pepper&Carrot to get the new PNGs..."
that reminds me of when i was trying to render an video, waiting 30 minutes for it to render, only to then find an issue with it.
then i figure out something obvious: i should test first with an small video then using what i really want to render, instead of waiting housr between every test.


omg, this gif is quite cool, the animated sound is not on purpose but its a nice effect

" (and also loosing any passion for Pepper&Carrot!)"
omg ! can understand you, that is sad/painfull to hear, but at least now i know i'm not the only one who lost the passion for my own projects due to the issues along the road.


" . This makes my renderfarm script now super hard to install because you need this rigg of special Inkscape versions and having multiple versions of the same software on GNU/Linux is hard. "
the most painfull of it is when you try to point out the problem to the comunity and they isist that you are wrong and the software is perfect, so you should be doing something wrong instead of the software, believe it or not, i saw that a lot.



" where the father attempt to "Replace a lightbulb" ( Imgur gifv link ). "
this is my everyday, lol

link   David Revoy   Author , - Reply

Thank you Igor for your comments! They made me smile. I guess we are a lot in the same boat :-)

link   Victor Cavalcanti   , - Reply

David, how do you feel about Krita's own text tools right now? They've improved somewhat from when I first started using it (at 2.8 era), and I find them to be usable most of the time, so I ditched Inkscape/Illustrator for the lettering part, because that extra step took me quite some time.

Also, amazing work on solving all of this stuff, even if it's not the perfect workaround. For someone like me who can't even code CSS with ease, that sounds incredibly hard to pull off!

And thank you always for Pepper & Carrot <3

link   David Revoy   Author , - Reply

Thank you Victor!
The built-in text-tool in Krita is receiving a total rewrite ( to SVG vector) and had minimal maintenance since years; but I still use it also here and there. Even today for the labels of the top picture here: http://www.davidrevoy.com/article330/ma … ing-system ; vector shape + titles. But I often limit it to a single line of text. It's ok also for small paragraph but a bit more unstable on my side. For Pepper&Carrot ; we can't work at 38 translating the same Krita file ; the file sharing would weight a lot while it's very lightweight to share only the SVG and version control system handle better the XML structure of the SVG, managing changes like code edition. If you read the report of a change on Github ; it's very easy to read it this way : https://github.com/Deevad/peppercarrot_ … 96bb6de9ff . With Krita files it would be more difficult to get this type of output.
Next Krita text tool and SVG layer will make me change workflow : I'll be able to letter my page in Krita and then export the layer to SVG Inkscape directly for the translation system. This will change my life, especially to prototype storyboard text with the good font size and feedback about how much room the text will eat in the page.
Thanks again for your comment.

link   Ted Henriksson   , - Reply

Regarding multiple installed version of software in *nix involvements, in my line of work (HPCC systems) we use http://modules.sourceforge.net/ to handle multiple versions being installed at the same time.

It's mostly used for compilers and other cli applications but I have used it for a wide variety of software with out problems. I know it won't solve any of the real underlying issues but it should make setup and scripting easier.

And thank you for Pepper & Carrot.

link   David Revoy   Author , - Reply

Thank you for pointing this technology ; I'll have a look at it!

link   Samuel Dell   , - Reply

This is one of the best articles I have come across. Keep up the good work.

Leave a reply

Notice : URL allowed ( auto-clickable ). Pictures URL too ( gif,jpg,png auto-displayed )

Capcha

Enter image code