[SPARKNLP-1079] AutoGGUFVisionModel

[DevinTDHa]

Description

This PR introduces the AutoGGUFVisionModel, enabling multimodal inference of llama.cpp GGUF models. The default pretrained model is llava_v1.5_7b_Q4_0_gguf_en.

Users can provide an image column and a text column with captions. The output will be a document column with the completions generated by the multimodal LLM.

Note that this annotator does not use the usual byte format provided by the Spark image reader (openCV compatible format). Instead, it requires raw image file bytes (e.g. from a jpg or png) which will then be decoded in the backend. For convenience, this is now implemented as a function of the ImageAssembler called ImageAssembler.loadImagesAsBytes.

For example:

Click to expand Example

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> from pyspark.sql.functions import lit
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("caption") \
...     .setOutputCol("caption_document")
>>> imageAssembler = ImageAssembler() \
...     .setInputCol("image") \
...     .setOutputCol("image_assembler")
>>> imagesPath = "src/test/resources/image/"
>>> data = ImageAssembler \
...     .loadImagesAsBytes(spark, imagesPath) \
...     .withColumn("caption", lit("Caption this image.")) # Add a caption to each image.
>>> nPredict = 40
>>> model = AutoGGUFVisionModel.pretrained() \
...     .setInputCols(["caption_document", "image_assembler"]) \
...     .setOutputCol("completions") \
...     .setBatchSize(4) \
...     .setNGpuLayers(99) \
...     .setNCtx(4096) \
...     .setMinKeep(0) \
...     .setMinP(0.05) \
...     .setNPredict(nPredict) \
...     .setNProbs(0) \
...     .setPenalizeNl(False) \
...     .setRepeatLastN(256) \
...     .setRepeatPenalty(1.18) \
...     .setStopStrings(["</s>", "Llama:", "User:"]) \
...     .setTemperature(0.05) \
...     .setTfsZ(1) \
...     .setTypicalP(1) \
...     .setTopK(40) \
...     .setTopP(0.95)
>>> pipeline = Pipeline().setStages([documentAssembler, imageAssembler, model])
>>> pipeline.fit(data).transform(data) \
...     .selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result") \
...     .show(truncate = False)
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|image_name       |result                                                                                                                                                                                        |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|palace.JPEG      |[ The image depicts a large, ornate room with high ceilings and beautifully decorated walls. There are several chairs placed throughout the space, some of which have cushions]               |
|egyptian_cat.jpeg|[ The image features two cats lying on a pink surface, possibly a bed or sofa. One cat is positioned towards the left side of the scene and appears to be sleeping while holding]             |
|hippopotamus.JPEG|[ A large brown hippo is swimming in a body of water, possibly an aquarium. The hippo appears to be enjoying its time in the water and seems relaxed as it floats]                            |
|hen.JPEG         |[ The image features a large chicken standing next to several baby chickens. In total, there are five birds in the scene: one adult and four young ones. They appear to be gathered together] |
|ostrich.JPEG     |[ The image features a large, long-necked bird standing in the grass. It appears to be an ostrich or similar species with its head held high and looking around. In addition to]              |
|junco.JPEG       |[ A small bird with a black head and white chest is standing on the snow. It appears to be looking at something, possibly food or another animal in its vicinity. The scene takes place out]  |
|bluetick.jpg     |[ A dog with a red collar is sitting on the floor, looking at something. The dog appears to be staring into the distance or focusing its attention on an object in front of it.]              |
|chihuahua.jpg    |[ A small brown dog wearing a sweater is sitting on the floor. The dog appears to be looking at something, possibly its owner or another animal in the room. It seems comfortable and relaxed]|
|tractor.JPEG     |[ A man is sitting in the driver's seat of a green tractor, which has yellow wheels and tires. The tractor appears to be parked on top of an empty field with]                                |
|ox.JPEG          |[ A large bull with horns is standing in a grassy field.]                                                                                                                                     |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------

A full end to end example of how to use this annotator is available at https://github.com/DevinTDHa/spark-nlp/blob/feature/SPARKNLP-1079-AutoGGUFVisionModel/examples/python/llama.cpp/llama.cpp_in_Spark_NLP_AutoGGUFVisionModel.ipynb

Motivation and Context

Enables users to caption images using multimodal LLMs.

How Has This Been Tested?

Tested the scala, python side locally, on databricks and colab.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.