LabelMe App Released: Dec 10, 2012 Version: 1.0 For iPhone and iPad.

The LabelMe App will allow you taking pictures, adding bouding boxes and uploading them your LabelMe online account.

A quick guide

Step 1: Get the LabelMe App

The LabelMe app is available for free at the App Store

Step 2: Create an account

If you have already a LabelMe account, you can use the same username and password. You can also create a user account from the App and use it at the LabelMe website. Once you create an account you can log in and start taking pictures and store them online.

Step 3: Collect and annotate images with the App

The images you collect will be uploaded to your online LabelMe account.

Step 4: Download the images and annotations

You can download the images from the LabelMe website (collection: iPhoneCollection).

You can also use the LabelMe toolbox to download the images and annotations. Use the matlab function:

APPdownload('yourUserName');

And that's it!

Some more details

The app has 3 sections: Labels Gallery, Detectors Gallery and Settings.

Labels Gallery

The Labels Gallery stores the images that you take and annotate. In this library you can see the images on a grid or a list layout. You can use the gallery view to upload and delete images.

The circle on the top-right corner of each image indicates if the image has been uploaded to the server. A number indicates how many new objects have been added since the last upload.

The gallery only shows the images available on the mobile device.

Image annotation

You can add bounding boxes around objects on each image. Once you have adjusted the bounding box to contain the object, you can add a tag to describe the object. You can use the keyboard or the microphone to tag the annotated object.

Once you upload the image you will be able to visualize the annotations on the LabelMe website. You can also make changes and add new annotations, however online changes will not be reflected on the data stores in the App and might get deleted if the image is uploaded again from the App.

Upload images

Images are uploaded to your account on the labelMe server. The images will be stored into the folder /iPhoneCollection/.

Currently, all the images captured from the App will be stored in a single folder. It is not possible to create collections within the App.

Downloading images

To download the images captured with the app you need to first upload them to the server and then download them from the LabelMe website.

Sharing images

If you share your username with other people, they will be able to download your images and annotations using the LabelMe toolbox.

If you want multiple people to help you collect and annotate images, they can create their own accounts and share the username with you so that you can download the images and annotations.

Delete images

If you delete images from the app, the images and annotations will only be deleted on the mobile device and will still be available on the LabelMe website if they have been uploaded. If you want to delete the images from the server you have to log in into the LabelMe website and delete the images there.

Deleting images on the App will allow you freeing memory without loosing the images that had been uploaded.

If you want to delete images from the server too, you have to do it from the LabelMe website.

Detectors Gallery

The Detectors Gallery shows you the list of the trained detectors. A detector, after trained, let you detect objects through the iOS device camera, in real time.

Adding a detector

To create a detector you first have to click the '+' sign on the left top of the detectors gallery.

Then choose the class (or classes) for you would like to create a detector. This classes are extracted from the Labels gallery, so you must have annotated images in order to train a new detector. Click next to continue.

The next screen shows you the available images to train from. This are all the annotated images you have from the Labels Gallery for the selected class. The default settings makes you select all the images. It is important to have a reasonable number of annotated images to train a good detector (see training considerations for more details). Click next to begin with the training.

During the detector training you will see a debug page with information about the training process. This page basically shows statistics about the evolution of the detector training. Wait until the Cancel button changes its name to Done.

when the training it is done, you will be prompted to the detector information page. This page shows you information about the detector and two images representing the detector: one is the average image of all the training set images and the other is a representation of the detector weights in the HOG space. Click the play button to execute it!

Executing detectors

To execute detector go to the detail page of it and hit the play button. You will see the iOS device camera and if you point to the trained object it should be detected.

The sliding bar sets de detector accuracy. If you swipe it to the left, more detections (with less accuracy) will be made. The opposite will happen if you swipe it to the right.

The settings button let you show on the screen the current Frames Per Second (FPS) achieved during the detection. Depending on the parameters used for training, you will be able to increase the FPS rate.

The HOG settings let you turn the camera into the Histogram Of Gradients (HOG) space. This is the space used were the detector is trained and it can help you debug and improve the dector. See the N. Dalal and B. Triggs article for more details.

There is also the possibility to execute more than one detector at the same time. To do it, just go to the detectors collection gallery, tap the Edit button and select as many detectors as you would like to execute simultaneously. Then tap the Execute button.

Retraining a detector

When a detector is created you can retrain it as many times as you want. This is useful, for example, when you want to improve the detector adding more images and annotations.

To retrain the detector, just go to the detector details and click the train button. You will have to choose again which training set you would like to use. Notice that the selected images that will appear are those that you use in the previous training.

Also you have the undo button to return to the previous trained detector if it has not improved.

Deleting a detector

To delete detectors just click the Edit button in the detectors gallery and select those you would like to delete.

Training considerations

To ensure a good performance on the object detection phase, here are some guidelines to take into account when training the detector.

• Annotation Size. To train the detector for a specific object, you have to take a picture of it ocupying the whole screen when inside the camera view. You will notice that when you open the picture though, there is extra space around the image. That is perfectly fine and indeed is it done to ensure a better training by learning of the object surroundings.
• Shape. The ideal shape is as squared as possible. LabelMe will also be able to detect very rectangular shapes, howevever it will be less accurate if those rectangular shapes vary accross the training set.
• Training set. The training set are all those annotated images you will use to train the detector. The performance of the detector is mostly affected by those images. Take pictures of the object to recognize with from all the perspectives you would like to recognize it from. Also take pictures in different enviroments so this wouls make the detecor more robuts. The number of images needed for the detector depends mostly on how complex is the object (different perspectives) and how robust you want it to be to different change of enviromets. Usual numbers are around 20 images per detector. For example, a chair is more complex thant a sphere, as it has more different perspectives. If I want the detector to recognize all possible chairs, I will also need to provide annotated images for different kind of chairs for different perspectives.
• Hog size. You can ajust the HoG size in the settings. This parameter let you tune the detector tradeoff between accuracy and speedness. More dimension to HOG means that the detector will be more accurate but it also will be slower.

Settings

There are several settings that can be modified:

• Save to library: if on the images will be saved to the iPhone camera roll.
• Image resolution: you can chose the image resolution. It is set to max resolution by default.
• GPS: you have to habilitate the location services in order to record the GPS coordinates inside the XML annotation file.
• Hog size: see the reference in the previous section.

Licenses

Please make sure that the images you upload are copyright-free and do not contain inappropriate content. The images that you upload and annotate will be considered part of the public domain and other people might see them. As the goal of LabelMe is to provide a tool for research, the images and annotations are expected to become available to the research comunity without restrictions.

You are free to post your collected database of images and annotations on your own website. For example, you may do this when you release your database with your research publications.

Privacy

Once you upload your images, they will be available online. Even if they are not listed, knowing the URL will be enough to access them. This makes easier to do research with the images collected.

Therefore, do not upload personal material that you do not want others to see

Citation

If you use the dataset or any functions on this website, we would appreciate it if you cite:
B. C. Russell, A. Torralba, K. P. Murphy, W. T. Freeman, LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, pages 157-173, Volume 77, Numbers 1-3, May, 2008. (paper.pdf)

Suggestions

Send email to labelme@csail.mit.edu if you have suggestions, find bugs or there are new features that you would like to see in the tool.

Credits

The LabelMe app has been developed by Dolores Blanco, Aina Torralba, David Way, Josep Marc Ming0t and Antonio Torralba.