Understanding the Product Import¶

It’s a good start to understand the overall architecture and how to re-use or replace some parts. You can now natively import data into CSV and XLSX format.

Note

The import part has been widely re-worked in 1.6 version of the PIM. Since 1.6 the old import system has been removed, please refer to previous versions of this page if needed.

Definition of the Job¶

Take a look at this configuration based on ConnectorBundle (src/Akeneo/Pim/Enrichment/Bundle/Resources/config/jobs.yml).

parameters:
    pim_connector.job.simple_job.class: Akeneo\Tool\Component\Batch\Job\Job

services:
    ## CSV product import
    pim_connector.job.csv_product_import:
        class: '%pim_connector.job.simple_job.class%'
        arguments:
            - 'csv_product_import'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            -
                - '@pim_connector.step.charset_validator'
                - '@pim_connector.step.csv_product.import'
                - '@pim_connector.step.csv_product.import_associations'
        tags:
            - { name: akeneo_batch.job, connector: 'Akeneo CSV Connector', type: 'import' }

    ## XLSX product import
    pim_connector.job.xlsx_product_import:
        class: '%pim_connector.job.simple_job.class%'
        arguments:
            - 'xlsx_product_import'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            -
                - '@pim_connector.step.charset_validator'
                - '@pim_connector.step.xlsx_product.import'
                - '@pim_connector.step.xlsx_product.import_associations'
        tags:
            - { name: akeneo_batch.job, connector: 'Akeneo XLSX Connector', type: 'import' }

With the type parameter, we can see that this job is an import.

Charset Validation Step¶

The purpose of this step is to validate that the input file has the expected encoding (default: UTF-8).

This step is a custom step, not a default ItemStep.

This step is defined in src/Akeneo/Tool/Bundle/ConnectorBundle/Resources/config/steps.yml

parameters:
    pim_connector.step.validator.class: Akeneo\Tool\Component\Connector\Step\ValidatorStep

services
    pim_connector.step.charset_validator:
        class: '%pim_connector.step.validator.class%'
        arguments:
            - 'validation'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            - '@pim_connector.validator.item.charset_validator'

We can also see that we inject a service pim_connector.validator.item.charset_validator in this step.

This service is defined in src/Akeneo/Tool/Bundle/ConnectorBundle/Resources/config/items.yml.

parameters:
    pim_connector.validator.item.charset_validator.class: Akeneo\Tool\Component\Connector\Item\CharsetValidator

services:
    pim_connector.validator.item.charset_validator:
        class: '%pim_connector.validator.item.charset_validator.class%'

The constructor of the CharsetValidator shows that it’s configured to check only a file which matches some extensions and to check the ‘UTF-8’ encoding.

/**
 * @param array  $whiteListExtension
 * @param string $charset
 * @param int    $maxErrors
 */
public function __construct(array $whiteListExtension = ['xls', 'xslx', 'zip'], $charset = 'UTF-8', $maxErrors = 10)
// ...

You can define your own service with the same class to validate other kinds of files or encodings.

As it implements Akeneo\Tool\Component\Batch\Step\StepExecutionAwareInterface, the step execution will be injected and usable during the execution.

The Akeneo\Tool\Component\Batch\Model\StepExecution allows to add information, messages and counters during the execution.

// for instance, add an info message when the check is not performed
$this->stepExecution->addSummaryInfo(
    'charset_validator.title',
    'job_execution.summary.charset_validator.skipped'
);

Note

This charset validator step can be re-used in other jobs (we use it in all file imports).

Note

The parsing of the bath_jobs.yml is quite specific, you can take a look at this class to understand it Akeneo\Tool\Bundle\BatchBundle\DependencyInjection\Compiler\RegisterJobsPass.

Product Import Step¶

The purpose of this step is to read an input file, to transform lines into product objects, to validate and save them in the PIM.

This step is a default step, an Akeneo\Tool\Component\Batch\Step\ItemStep is instantiated and injected.

parameters:
    pim_connector.step.item_step.class: Akeneo\Tool\Component\Batch\Step\ItemStep

services:
    pim_connector.step.csv_product.import:
        class: '%pim_connector.step.item_step.class%'
        arguments:
            - 'import'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            - '@pim_connector.reader.file.csv_product'
            - '@pim_connector.processor.denormalization.product'
            - '@pim_connector.writer.database.product'

pim_connector.step.xlsx_product.import:
    class: '%pim_connector.step.item_step.class%'
    arguments:
        - 'import'
        - '@event_dispatcher'
        - '@akeneo_batch.job_repository'
        - '@pim_connector.reader.file.xlsx_product'
        - '@pim_connector.processor.denormalization.product'
        - '@pim_connector.writer.database.product'

An ItemStep always contains 3 elements, a Akeneo\Tool\Component\Batch\Item\ItemReaderInterface, a Akeneo\Tool\Component\Batch\Item\ItemProcessorInterface and a Akeneo\Tool\Component\Batch\Item\ItemWriterInterface.

We provide here specific implementations for these elements, the services are declared with aliases pim_connector.processor.denormalization.product.flat.

Product Reader¶

This element reads a file and converts items one by one into standard format (it indexes each line with field names).

[
    'sku'           => [
        ['data' => 'AKNTS_BPXS', 'locale' => null, 'scope' => null]
    ],
    'categories'    => ["goodies", "tshirts"],
    'clothing_size' =>
        [
            [
                'locale' => NULL,
                'scope'  => NULL,
                'data'   => 'xs',
            ]
        ],
    'description' =>
        [
            [
                'locale' => 'en_US',
                'scope'  => 'mobile',
                'data'   => 'Akeneo T-Shirt'
            ],
        ],
]

The service is defined in src/Akeneo/Pim/Enrichment/Bundle/Resources/config/readers.yml.

parameters:
    pim_connector.reader.file.xlsx_product.class: Akeneo\Pim\Enrichment\Component\Product\Connector\Reader\File\Xlsx\ProductReader
    pim_connector.reader.file.csv.class: Akeneo\Tool\Component\Connector\Reader\File\Csv\Reader

services:
    # CSV Reader
    pim_connector.reader.file.csv_product:
        class: '%pim_connector.reader.file.csv_product.class%'
        arguments:
            - '@pim_connector.reader.file.csv_iterator_factory'
            - '@pim_connector.array_converter.flat_to_standard.product_delocalized'
            - '@pim_connector.reader.file.media_path_transformer'

    # XLSX Reader
    pim_connector.reader.file.xlsx_product:
       class: '%pim_connector.reader.file.xlsx_product.class%'
       arguments:
           - '@pim_connector.reader.file.xlsx_iterator_factory'
           - '@pim_connector.array_converter.flat_to_standard.product_delocalized'
           - '@pim_connector.reader.file.media_path_transformer'

Note

This step is able to extract a zip archive which contains a file for products and next to it a folder containing images. The product file refers to images using relatives paths.

StandardArrayConverterInterface¶

This service allows to transform the CSV array of items to the Standard Format array.

// CSV Format
$csvItem = [
  'sku'                         => 'AKNTS_BPXS'
  'categories'                  => 'goodies,tshirts'
  'clothing_size'               => 'xs',
  'description-en_US-mobile'    => 'Akeneo T-Shirt',
  'description-en_US-ecommerce' => 'Very Nice Akeneo T-Shirt',
];

$standardItem = $this->arrayConverter->convert($csvItem);

// Standard Format
[
    'sku'           => [
        ['data' => 'AKNTS_BPXS', 'locale' => null, 'scope' => null]
    ],
    'categories'    => [ 'goodies', 'tshirts'],
    'clothing_size' => [
        ['data' => 'xs', 'locale' => null, 'scope' => null]
    ]
    'description'   => [
        ['data' => 'Akeneo T-Shirt', 'locale' => 'en_US', 'scope' => 'mobile'],
        ['data' => 'Very Nice Akeneo T-Shirt', 'locale' => 'en_US', 'scope' => 'ecommerce'],
    ]
]

AttributeLocalizedConverterInterface¶

When you import a product with localized attributes (e.g. prices with comma as decimal separator), data will be converted to transform comma to dot.

$convertedItem = $this->convertLocalizedAttributes($convertedItem);

The service uses the class Akeneo\Tool\Component\Localization\Localize\AttributeConverter.

Note

Read the cookbook to add your own localizer How to Localize your data

Product Processor¶

This element receives items one by one, creates (or fetches if it already exists) the related product, updates and validates it.

The service is defined in src/Akeneo/Pim/Enrichment/Bundle/Resources/config/processors.yml.

parameters:
    pim_connector.processor.denormalization.product.class: Akeneo\Pim\Enrichment\Component\Product\Connector\Processor\Denormalizer\ProductProcessor

services:
    pim_connector.processor.denormalization.product:
        class: '%pim_connector.processor.denormalization.product.class%'
        arguments:
            - '@pim_catalog.repository.product'
            - '@pim_catalog.builder.product'
            - '@pim_catalog.updater.product'
            - '@pim_catalog.validator.product'
            - '@akeneo_storage_utils.doctrine.object_detacher'
            - '@pim_catalog.comparator.filter.product'
            - '@pim_catalog.localization.localizer.converter'

The class Akeneo\Pim\Enrichment\Component\Product\Connector\Processor\Denormalizer\ProductProcessor mainly delegates the operations to different technical and business services.

/**
 * @param IdentifiableObjectRepositoryInterface $repository    product repository
 * @param ProductBuilderInterface               $builder       product builder
 * @param ObjectUpdaterInterface                $updater       product updater
 * @param ValidatorInterface                    $validator     product validator
 * @param ObjectDetacherInterface               $detacher      detacher to remove it from UOW when skipping an item
 * @param ProductFilterInterface                $productFilter product filter
 */
public function __construct(
    IdentifiableObjectRepositoryInterface $repository,
    ProductBuilderInterface $builder,
    ObjectUpdaterInterface $updater,
    ValidatorInterface $validator,
    ObjectDetacherInterface $detacher,
    ProductFilterInterface $productFilter
) {
    // ...
}

IdentifiableObjectRepositoryInterface¶

This service allows to fetch a product by its identifier (SKU by default).

$product = $this->repository->findOneByIdentifier($identifier);

This is possible because the Akeneo\Pim\Enrichment\Bundle\Doctrine\ORM\Repository\ProductRepository implements Akeneo\Tool\Component\StorageUtils\Repository\IdentifiableObjectRepositoryInterface.

ProductBuilderInterface¶

If the product doesn’t exist yet, we use this service to create one with provided identifier and family code.

$product = $this->builder->createProduct($identifier, $familyCode);

The service uses the class Akeneo\Pim\Enrichment\Component\Product\Builder\ProductBuilder.

ProductFilterInterface¶

When a product already exists, this service allows to normalize the current product data to the Standard Format array.

Then, it compares the current data against the updated data provided by the StandardArrayConverterInterface to present only new or changed values.

This comparison mode can be enabled or disabled with the configuration parameter enabledComparison of the product import.

$filteredItem = $this->filterIdenticalData($product, $convertedItem);

The service uses the class Akeneo\Pim\Enrichment\Component\Product\Comparator\Filter\ProductFilter.

Note

This parameter can have a large impact on the performance when it’s enabled.

When your import handles a file of existing products with a lot of columns but few updated values, it may divide the execution time by ~2.

When your import handles a file of existing products when all values are changed, it may cause an overhead of ~15%.

Don’t hesitate to test and use different configurations for different product imports.

ObjectUpdaterInterface¶

Once fetched or created, this service allows to apply changes to the product.

The format used by the update method is the Standard Format array.

An important point to understand is that the modifications are applied only in memory, nothing is saved to the database yet.

$this->updater->update($product, $filteredItem);

The service uses the class Akeneo\Pim\Enrichment\Component\Product\Updater\ProductUpdater.

ValidatorInterface¶

Once updated, the product is validated by this service.

This service uses Symfony\Component\Validator\Validator\ValidatorInterface.

$violations = $this->validator->validate($product);

If violations are encountered, the product is skipped and the violation message is added to the execution report.

When an item is skipped, or not returned by the processor, the writer doesn’t receive it and the item is not saved.

if ($violations->count() > 0) {
    $this->detachProduct($product);
    $this->skipItemWithConstraintViolations($item, $violations);
}

Note

You can notice here a very specific usage of the ObjectDetacherInterface, it allows to detach the product from the Doctrine Unit Of Work to avoid issues with skipped products and the ProductAssociation Step.

This detach operation is not the responsibility of the processor and the usage here is a workaround.

Product Writer¶

This element receives the validated products and saves them to the database.

The service is defined in src\Akeneo\Tool\Bundle\ConnectorBundleBundle\Resources\config\writers.yml.

parameters:
    pim_connector.writer.database.product.class: Akeneo\Pim\Enrichment\Component\Product\Connector\Writer\Database\ProductWriter

services:
    pim_connector.writer.database.product:
        class: '%pim_connector.writer.database.product.class%'
        arguments:
            - '@pim_versioning.manager.version'
            - '@pim_catalog.saver.product'
            - '@akeneo_storage_utils.doctrine.object_detacher'

The class Akeneo\Pim\Enrichment\Component\Product\Connector\Writer\Database\ProductWriter mainly delegates the operations to different technical and business services.

/**
 * Constructor
 *
 * @param VersionManager              $versionManager
 * @param BulkSaverInterface          $productSaver
 * @param BulkObjectDetacherInterface $detacher
 */
public function __construct(
    VersionManager $versionManager,
    BulkSaverInterface $productSaver,
    BulkObjectDetacherInterface $detacher
) {
    // ...
}

BulkSaverInterface¶

This service allows to save several objects to the database at once.

For products, the implementation of Akeneo\Pim\Enrichment\Bundle\Doctrine\Common\Saver\ProductSaver is used.

A dedicated chapter explains how it works How to Save Products.

BulkObjectDetacherInterface¶

This service allows to detach several objects from the Doctrine Unit Of Work at once to avoid keeping them in memory.

In other terms, it avoids keeping all the processed objects in memory.

Product Association Import Step¶

Once the products are imported, this step allows to handle associations between products.

We use a dedicated step to be sure that all valid products have already been saved when we link them.

The purpose of this step is to read the input file, to transform lines to product association objects, and to validate and save them in the PIM.

This step is a default step, an Akeneo\Tool\Component\Batch\Step\ItemStep is instantiated and injected.

services:
    ## CSV Import
    pim_connector.step.csv_product.import_associations:
        class: '%pim_connector.step.item_step.class%'
        arguments:
            - 'import_associations'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            - '@pim_connector.reader.file.csv_association'
            - '@pim_connector.processor.denormalization.product_association'
            - '@pim_connector.writer.database.product_association'
            - 1

    ## XSLX Import
    pim_connector.step.xlsx_product.import_associations:
        class: '%pim_connector.step.item_step.class%'
        arguments:
            - 'import_associations'
            - '@event_dispatcher'
            - '@akeneo_batch.job_repository'
            - '@pim_connector.reader.file.xlsx_association'
            - '@pim_connector.processor.denormalization.product_association'
            - '@pim_connector.writer.database.product_association'
            - 1

We provide here specific implementations for these elements, the services are declared with aliases pim_connector.reader.file.csv_association, pim_connector.processor.denormalization.product_association, pim_connector.writer.database.product_association.

This step is composed of quite similar parts of the product import step but relatively more simple because it handles fewer use cases.

Found a typo or a hole in the documentation and feel like contributing?
Join us on Github!