Category: Projects

Booktype Easy Install

aerkalov 5 Comments

A lot of people seems to have problems installing Booktype. Until we have native packages for each distribution installing vanilla Booktype from git repository will be the way to go.

I love all of my users the same and that is why I decided to work on a small install script which would do most of the steps for them. This script should help you even when your distribution does not provide valid native packages (Old Django packages and etc…). Anyhoo, I have done some basic testing on – Ubuntu 10.04, Ubuntu 12.04, CentOS 6.3, Debian 6 and I have tried to avoid combination of outdated system packages and latest packages installed with PIP (because that is what Google told you to do). I am aware there are many places where this script could go wrong but i have tried to fix at least basic problems. For instance, if you are on CentOS it will warn you and inform how to install EPAL repository.

Like all “double click” methods this script will also install unwanted packages. To avoid conflicts with different versions of python modules it will create virtualenv environment and install only those Python packages (and their versions) you really need. It will install packages using python package tools and not system packages. For some of the packages it means compiling. Compiling means having development tools on the system. Be aware of that.

Created Booktype instance will use Sqlite to store data. Sqlite is great database but it does not support easy upgrades of schema. If you are serious about Booktype you might want to look at PostgreSQL. After all, this is version 0.0.1 and we only support built in python web server. To make it work with Apache you will need to manually activate virtualenv environment in wsgi script. In the future install script will do this for you.

Where to get it

For now, Booktype Easy Install script is part of “Booktype Scrolls” project and you can get it here:
https://github.com/aerkalov/booktype-scrolls/tree/master/scripts/install.

How to start it

Download the script and execute it. It will download and install required packages. When it is done it will tell you how to create user account and start Booktype. You don’t need to be root to start this install script but you do need sudo permissions in case it must install some system packages. You will need to confirm installation of new packages and you will be informed which commands are being executed in the background. Without your permission new packages will not be installed on the system. Feel free to analyze the script before you start it.

wget https://raw.github.com/aerkalov/booktype-scrolls/master/scripts/install/booktype_install.py 
python booktype_install.py

Use –help to get all available arguments:

python booktype_install.py --help
python booktype_install.py -p myproject
python booktype_install.py -o ubuntu -p book

What now

It needs testing and support for more platforms. Please report to me if you have problems with it. Will try to fix it together!

Booktype scrolls – Importing WordPress site!

aerkalov Leave a comment

So…you left your job and went traveling throughout Asia for 4 months.  In small cyber cafe, somewhere in south Pakistan, you created WordPress site because you were lazy to send emails to friends and family. Internets loved you and your pictures were on the front page of Reddit many many times. Who could forget that great article about food poisoning in China or that unfortunate misunderstanding with gentleman in Bangkok…

When you got back everyone was like – “You should write a book!” and you said to yourself  – “Why not… Half of the book is already written.”. This brings us to the subject of this post – importing WordPress site as a Booktype book.

Importing

There are couple of ways you can do it (this can be applied to other systems also):

  • Use some kind of API to remotely access data. For this you only need permission to access data.
  • Write your own plugin to export data from WordPress. Besides knowing how to write plugin you need permissions to install it on the remote server. You can use WordPress API to render the articles properly, something you can’t really do with other 2 methods.
  • Finally the method we are using in this post. Parsing XML export file created with WordPress export plugin.

Importing data can be complex process and each method has its good and bad points. You probably don’t want to have chapter for each post in WordPress because that could be hundred and hundred of chapters at the end. Maybe you want to choose which chapters you want and sometimes to combine couple of posts as one chapter. These are just some of the issues.

How it works

As it is visible on the picture above you should go to Tools/Export and choose what you want to export. Download the “Export File” to you computer. Export file is just extended RSS file and you can (and should) use it to export content of your site. Notice that attachments and images are not part of this export file, we will need to download them separately! Besides content is in “raw” format (before everything was rendered to HTML).

Like I said, export file is extended RSS file and for parsing we use this great library feedparser. Book name can be specified with arguments but for the default name we use title of the WordPress site. Booktype cares about two titles. One is the full title of the book and the other one is unique url name of the book (usually slugified version of full title).

from booki.utils.book import createBook

book = createBook(conf['user'],
                  conf['bookTitle'],
                  status = 'new',
                  bookURL=conf['bookTitleURL'])

As you can see we use function createBook. It takes user object as first argument (someone has to be owner of the book), full book title, default book status (books can have different statuses) and slugified title name. Export file provides us with the information about WordPress administrator and Post author. We could use that information to find or create Booktype user with specific info and set that person as Book owner.

Parsing posts

After this we just go through the list of all the posts in export file and ignore those who don’t have ‘wp_status’ set ‘publish’. Like i said, content of that chapter is in “raw” format. For instance empty lines in text are representing start of new chapter and etc. If we were using specific tags to format our code (for instance, to prettify the source code) we would not get nice and colorful HTML.

In this example we just do two modifications. One is to create paragraphs in text and the 2nd one is to put Chapter title inside of H2 tag. That is one of the Booktype requirements at the moment.

content = "\n".join(["< p >%s< /p >" % p for p in content.split('\n\n') if p.strip() != ''])
content = u'< h2 >%s< /h2 >%s' % (chapterTitle, content)

And yes… I have some extra spaces in tags P and H2 because i am too lazy to figure out how to escape it in code prettifier plus the code to make paragraphs is not perfect (but this is just an example).

Images

Now we parse Post content and search for the images. When we find one, we try to download it to our computer. Normally we wouldn’t need to do it, but Booktype really wants to have entire content of a book locally. When the image is successfully downloaded we save it as an attachment and we modify image location. All images must be placed inside of ‘static/’ directory. There are good reasons why it has to be relative path, but we will not talk about it now.

att = models.Attachment(book = book,
         version = book.version,
         status = stat)


f2 = File(StringIO(data))
f2.size = len(data)
att.attachment.save(fileName, f2, save=True)


fileName = os.path.basename(att.attachment.path)


e.set('src', 'static/%s' % fileName)

Chapter

At the end we just create new Chapter. This is very basic way of creating new chapter because we are not leaving any traces in our logs and etc… We can leave that for some other example. Also notice we are not doing anything with links to other posts/chapters on the same site/book. We just ignore that for now.

chapter = models.Chapter(book = book,
                         version = book.version,
                         url_title = bookiSlugify(chapterTitle),
                         title = chapterTitle,
                         status = stat,
                         content = content,
                         created = now,
                         modified = now)


chapter.save()

Source

Look at the full source for WordPress importer in my Booktype Scrolls repository. Purpose of this code is to be more educational and less production ready.

As you can see technical part of importer is very simple. Thumblr has API for remote access and it would be fairly simple task to make Thumblr importer (or for any other CMS). The biggest problem is still how to make their HTML work nicely inside of Booktype. If author is using custom CSS on WordPress side then Booktype would be completely unaware of it. For instance – <img class=”alignnone size-full wp-image-591″ … It would be the same for different WordPress plugins, broken HTML, headings at wrong place, JavaScript plugins and etc.

Booktype Scrolls

aerkalov Leave a comment

I decided to put different kind of scripts, snippets and small Django apps in a separate project called “Booktype Scrolls“. The idea is to show how Booktype can be extended and how some of its functionality can be used.

For now i have WordPress importer, global search-replace and couple of snippets to get some basic statistics from the database. For 0.1.3 version it is more then satisfactory but I plan to extend it every couple of days with something new. So… stay tuned!

Links:

Booktype 1.5.3

aerkalov Leave a comment

Booktype 1.5.3 is out with a lot of bug fixes and new features. Check the full list of commits here – CHANGES.txt. As you could also notice we were a bit late with this release.

Major feature is something called “Booktype Control Center”. Default Django admin interface is still here but we have developed new administrative interface for Booktype administrators. Booktype is now more customizable then ever and  you  can use this web tool to adapt it to your own needs. Besides users and books you can also manage look and feel of your Booktype instance and books your produce. We started as extremely open tool for book collaboration but with this release we have let administrators adapt Booktype instance to their own needs. Normal users can not create books? Not a problem! Only administrator can create new users? It it possible now! You only want to let users output books as PDF files? You can do it now!

Because of “Booktype Control Center” we needed to develop new way of managing configuration. Read more about it in “Managing configuration” blueprint. Only some parts of code are using this new API but i plan to convert entire Booktype in new release (that would be 1.5.4). Besides us developers, new API will be very useful to administrators during migrations or daily Booktype management.

We have also upgraded our default settings.py to work with latest version of Django. Django 1.2 is still minimal requirement but now we work out of the box with Django 1.4! No more manual editing of files! This is a feature for new users, everyone else will have to manually update their old settings.py files. If it is too much work for someone (as always, INSTALL file has information how to do it) they can always create new Booktype project and just copy+update it with their existing project.

New release will be 1.5.4 (not that hard to guess) and it should be out in 2 weeks. The plan is to have release every 2 weeks. What can you expect in new release? More API cleaning, new install scripts and couple of secret projects. But nothing is a secret with Booktype, if you want to be up to date with our development watch our github page, check my tweets and read my blog!

Getting Started with Booktype – What is Booktype?

aerkalov Leave a comment

This is a first post in a series of articles about Booktype. Before I start to be all technical and “… and that is how you install new Django app” it would be good to say couple of words about Booktype history.

Booktype is an open source platform to write and publish print and digital books. It was created to run the FLOSS Manuals platform and back then it was just called Booki. FLOSS Manuals is a community of people joined together for their love of writing good quality documentation for FLOSS software. At first the  FLOSS Manuals platform was build on the  TWiki (Wiki in Perl) platform.

Using  TWiki to prototype something was a good choice. We extended it with our own plugins and some external tools. FLOSS Manuals is a collaborative community and we needed to extend TWiki to support the collaborative processes. We had our IRC-Web chat gateway called FM Transmission [post in Croatian], live Table of contents where you could see what other users are doing (view image of it), Remixer for content, Localization plugins to manage book translation, our own Web-based tool for translating content and many other things. It was good enough platform but it felt like a big hack at times. After some thinking and observing some problems in the TWiki community [post in Croatian] we decided to write a new platform –  from scratch.

We wanted to build community around software and we figured out TWiki and Perl were not the best platform for that. Mainly because a lot of our users are people who are more interested in books and less in Perl code. We decided to use Python and Django as our new platform. They both seems to be reasonably easy to master and even newbies could extend their Booktype installations. You want to have blog on the front page? Just install existing Django blog app, change one line in settings file, put couple of tags in templates and you are good to go. Simples! Anyone could do it.

Booktype

When we talk about Booktype we really talk about Booktype and Objavi. Booktype is used for editing and managing books while Objavi is used to produce book formats (as pdf, epub, mobi, …). In this post we will talk only about Booktype (the book editor).

Booktype is all about the collaborative editing of books and as such is not always what people want. It is about building community around your book. Sure, you can use it to write your diary also (and you should!), but the real power is in collaborative work. What does this mean for the architecture?

Like i said, we decided to use Django as web framework mainly because we wanted people to be able to plug Booktype into their existing web sites more easily and extend their installations with many already existing Django apps. I think we made a good choice with it. As database backend we are using PostgreSQL. Sqlite3 is also an option, but it really should be used just for testing and development purposes. We are using Redis in-memory database to store all kind of information web clients are exchanging with Booktype server using Sputnik.

What is Sputnik? Sputnik is a Django application for two-way communication with web clients. When we started there was no other Django project that could help us with this so I decided to do what everyone would normally do. I wrote one abstract API (with channels, subscriptions and etc…) for communication and implemented dummy network layer. It means when the right moments comes this network layer can be replaced with something better without any need to change the rest of the code. There are many good frameworks these days that could be used as network layer (Socket.IO for instance). Why do we have it? Mainly because of the collaborative part of Booktype. When you have 10 people working on one book at the same time you need a way of notifying all of them about the actions others are doing. Like: locking chapters, chat, changes in Table of Contents, creation of new chapters, attachments uploads and etc. Yes, it means your Table of Contents will be changed as soon as someone adds a new chapter to it. This really improves the user experience and stops you from overwriting someone’s changes.

But enough about Booktype history. In my next article I will talk about exciting world of “Booktype projects” and how you can easily install and extend your Booktype installation!

These are the Booktype links  you should probably check:

 

 

 

Franjo Kluz ponovo leti

aerkalov Leave a comment

Daklem, nabavio sam si HTC Desire Z. Kako to obično biva kod mene, otišao sam do VIP Centra nakon rasprave na poslu i rekao “hoću to”. Upiknuo sam skoro kao ćorava kokoš. Na “Ništa se neće dogoditi” u Moravicama sam ga malo držao u rukama, zapamtio da su rekli “to je najbolje što ima tastaturu” (valjda), proguglao 10 minuta i otišao kupiti.

Moj prvi utisak je da je to jedno obično govno i nisam siguran da bih to kupio roditeljima. Zglobovi koji drže tastaturu su se totalno pohabali nakon 2 dana korištenja. Super da sam guglao samo 10 minuta, jer su Interneti prepuni postova ljudi o ovom problemu. Novije serije su malo bolje ali mi smo izgleda u Hrvatska dobili stariju seriju. Brijem da nema teorije da će mi ovo zamjeniti. U svakom slučaju, svi su mi se igrali sa telefonom i namjerno forsirali taj zglobić. Da ga nisu drugi koristili, meni bi se to pohabalo nakon tjedan dva normalnog korištenja.

Drugi problem koji sam otkrio je da mu je trebalo izrazito dugo da pošalje SMS. Još jedan bug, ovaj put službeni koji je ispravljen u updateu. Dan i nešto sam koristio telefon i isčuđavao se sa “pa zar mu zaista treba minuta da pošalje sms?!”. Da… dragi moj čitaoče. U zavisnosti od položaja zvijezda na nebu, SMS bi slao od 10 sekundi do nekih minutu i nešto. Naravno, sve stoji dok on šalje SMS i vrti prokleti progres bar. Pa tko to pušta u prodaju?!

Sve je to ok, ali ovo je ipak potrošačka tehnika koja se kupuje na štandu. Ne bi trebalo imati ovakve probleme. Sa ovim svojim PC pristupom od 50 ne baš u potpunosti kompatibilnih hardware-skih platformi i 50 različitih verzija OS-a su me ipak uspjeli natjerati da nešto i napišem za Androida. Već imam osjećaj da će to biti veselo iskustvo. Isprobavao sam od frenda aplikaciju za upload videa. Naravno, kad ja otvorim svoju tastaturu u trenutku upisivanja naziva videa, njegova se aplikacija totalno skrši. Jelte… nije čovjek imao blizu sebe Androida sa tastaturom da isproba.

Daklem…. *taarraaaaam*. Počeo sam se malo igrati sa Franjom Kluzom (kojeg sam zapostavio ravno dvije godine) na Androidu. Malo prisjećanja Jave koju nisam dirao više od 10 godina, malo igranja sa Eclipseom, malo upoznavanja sa ostalim alatima, malo API-ja, …Ovo je neka moja ideja kako ja zamišljam intro screen u novom Franji (loše izgleda za sada, naravno trza itd…). Malo animacije, malo muzike i nekih poruka… Sve u svemu, još sam u procesu učenja uz standardne zanimacije na poslu + poslovima sa strane koji uvijek pate. E pa neka pate, jer za mjesec-dva Franjo Kluz sa jednim nivoom izlazi na Internete! Strepi Minecraftu, strepi!

Više o Franji na njegovom službenom siteu.

Glasajte za Booki!

aerkalov Leave a comment

Gdje to treba glasati ?

Transmediale je u suradnji sa Mozillom najavio Open Web Award 2011. Pročitajte više na website-u o čemu se radi.

Flossmanuals su uključeni sa projektom Booki. Ne trebam pričati koliko ću biti sretan kad bi glasali za nas i pomogli nam da se dokopamo 5000 EUR-a nagrade i vječne slave!

Što je Booki ?

Booki je naziv naše nove platforme za jednostavno i brzo objavljivanje knjiga. To uključuje softwarea koji služi za multiuser online editiranje knjiga, publish-anje knjiga u različite formate, importanje sadržaja sa drugih platformi itd… Osim samog software (platforme) to uključuje i Book sprintove koje organiziramo, a od skoro i kombi. Kombi? Oh da… Kombi sa opremom za printanje i uvezivanje koji je u stanju doći i razveseliti vaše lokalno događanje!

Zašto bi glasao/la za Booki?

Postoji nekoliko razloga zašto bi mogli glasati za nas!

Prvo razlog je što smo vam mi baš super! Super vam je ekipa uključena u projekt, super vam je ekipa koja je odlučila koristiti naš sustav za svoje projekte,… Npr. ekipa kao što je Arctic Perspective Initiative (http://arcticperspective.org) koji rade na knjizi o preživljavanju u Artiškim uvjetima, Thunderbird / Inkscape / Processing / CimCRM / Blender / Command line / OLPC i ostali manuali, ekipe koja skuplja folk priče o uzgajanju biljki, itd. Super vam je ekipa koja odvaja ono malo slobodnog vremena od svog posla i obitelji da potroši na stvari do kojih im je stalo, a vjerujte mi, takvih je dosta uključeno u projekt i stvari koje se dešavaju oko projekta. Oh… kako bi im samo Vas glas puno značio 🙂

Drugi razlog je što bi svaki vaš glas mene malo trgnuo i natjerao da mrdnem guzicu i više radim na projektu. U zadnje vrijeme sam se fino ulijenio. Što je naravno velika šteta jer se radi o jednom od najzanimljivijih projekata u mom dosadašnjem zivotu. Pa ako mislite da će Vaš glas pomoći da mrdnem guzicu od besmislenog komercijalnog posla koji ne pruža ništa osim stalnog prihoda, nemojte se previše sramiti da glasate za nas!

Kako glasati ?

Ako ste došli do ovog paragrafa, neće vam biti onda teško ni da se registrirate na websiteu i glasate za nas! Znam … znam.. ali tako stvari stoje 🙂

Prvo se registrirajte:
http://www.drumbeat.org/user/register

I onda glasajte za nas:
http://www.drumbeat.org/project/open-web-publishing

Hvala od mene i cijelog tima!

FLOSS Manuals – Theora Cookbook

aerkalov Leave a comment

Završio je četverodnevni Theora sprint u Berlinu. Rezultat je u vidu 212 stranica papira …. Theora Coockbook knjiga… ttarrraaaaa….. Sprint podržao Google i Berlin Summercamp.

Kao i obično, idućih par dana će se prolaziti kroz tekstove i ispravljati greške. Užasno mi je žao što na ovome nisam mogao biti… ali eto, teško je usaglasiti regularan posao sa paradiranjem po Evropi i sudjelovanju u različitim projektima. U ovaj manual se uspio progurati i Barack Hussein Obama. Skidajte, čitajte i širite glas dalje (naravno, možete i pomoći u pisanju).

Link na web izdanje:
http://en.flossmanuals.net/theoracookbook

Link na PDF i papirnato izdanje:
http://www.lulu.com/content/7531795

Link na FLOSS Manuals:
http://en.flossmanuals.net/

Conversation and Community: The Social Web for Documentation!

aerkalov Leave a comment

Anne Gentle piše za paru tehničku dokumentaciju. U slobodno vrijeme radi na FLOSS Manualsima, piše neke svoje priče i stara se o mužu i dvoje djece. Simpatična teksašanka (donijela mi je nekoliko kutijica Altoidsa). Teško da ću zaboraviti naš susret u Amsterdamu. Njen prvi put u Evropu i odmah poslom u Amsterdam. Smješak na licu koji kaže “ah da je sve ovo prije 15 godina”.

Daklem. Izdala je žena svoju prvu knjigu pod nazivom “Conversation and Community: The Social Web for Documentation”. Ako vas zanima ta tematika slobodno pogledajte free chapter (spominje se i moja malenkost unutra…. samo riječju i imenom ali eto).

Više o knjizi na:

Announcing Conversation and Community: The Social Web for Documentation!

Friedrich Nietzsche i thumbnailovi sa images.google.com

aerkalov Leave a comment

Izgleda da je ovo tjedan 5 lajnera u pythonu. Anywho, radim neki backend za jedan hax0rski portal. U biti, radim neuspješnu reinkarnaciju Metafeeda-a u vidu naslovnice za site Razmjena vještina. Svečana prezentacija bi trebala biti na “Ništa se neće dogoditi”. Nakon toga siguran sam da će to biti vaš prvi (ako ne i jedini) feed uz koji ćete ujutro pijuckati kavu.

Daklem… jedna od stvari koja mi treba za backend portala je operacija “uzmi mi random sliku sa images.google.com”. Mislio sam da će biti nekog HTML parsanja sa BeautifulSoup ali moj dragi Google me lijepo iznenadio. Pljunu natrag gomilu JavaScripta koji izgenerira stranicu u samom Browseru. Koristeći najobičniji regular expression uzmem sadržaj tog JavaScript Arraya (kao string). Zgodno je što je u ovom slučaju sintaksa za Array polja u JavaScriptu identična Pythonovim listama pa iskoristim običan Pythonov eval da od toga dobijem Pythonovu listu. Da sam i imao želju za Djeda Mraza (iliti lika sa Coca-Coline reklame) ne bi bilo ovako lako na kraju.

Naravno, cijela ova fora će raditi dok Google ne odluči promjeniti sintaksu. Cilj skripte je pokazati ukratko kako sam rješio problem dobivanja URL-a slike (a i da popunim blog ovom trivijalijom). Zbog toga i nisam onečistio code sa provjerama u slučaju grešaka i neispravnih rezultata. Baš mi nešto i ne radi num argument za podešavanje broja rezultata na stranici. Zato koristim argument -p koji mi kaže na koju stranicu rezultata da odem.

Upotreba:

skini.py -p 4 krava muzara
skini.py -r svinjska gripa

Skripta skini.py: 

#!/usr/bin/python


import urllib2, urllib, re, sys, getopt, random



def fetchURL(query, start = 0):
    req = urllib2.Request('http://images.google.com/images?hl=en&q=%s&gbv=2&aq=f&oq=&aqi=g10&start=%s' % (urllib.quote_plus(query), start))

    req.add_header('User-Agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.1pre) Gecko/20090701 Ubuntu/8.10 (intrepid) Shiretoko/3.5.1pre')
    req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')
    req.add_header('Accept-Language', 'en-us,en;q=0.5')
    req.add_header('Accept-Encoding', 'deflate')
    req.add_header('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7')
    req.add_header('Connection', 'close')

    r = urllib2.urlopen(req)
    
    data =  r.read()

    return data



def parseImages(data):
    p = re.compile(";dyn\.setResults\(\[(.+)\]\);\<\/script\>")
    m = p.search(data)
    lst = eval(m.group(1))

    return lst



if __name__ == '__main__':
    try:
        optlist, args = getopt.getopt(sys.argv[1:], 'rp:')
    except getopt.GetoptError, err:
        print "skini.py [-p <page number>] [-r] args\n"
        print str(err)
        sys.exit(-1)

    start, isRandom = 0, False

    for arg, value in optlist:
        if arg == '-p': start = value
        if arg == '-r': isRandom = True

    data = fetchURL(' '.join(args), start = start)
    images = parseImages(data)

    if not isRandom:
        for img in images:
            print img[3]
    else:
        print random.choice(images)[3]