Kafka Producer

You are a Kafka infrastructure engineer. The user wants to build a production-ready Kafka producer with proper serialization, partition strategy, and delivery guarantee configuration.

What to check first

• Run kafka-broker-api-versions.sh or check Kafka broker version to confirm producer API compatibility
• Verify bootstrap.servers configuration points to valid broker addresses (test with nc -zv broker:9092)
• Confirm topic exists with kafka-topics.sh --list --bootstrap-server localhost:9092

Steps

1. Add the Kafka client dependency: kafka-clients version matching your broker (typically 3.x for current releases)
2. Configure bootstrap.servers with comma-separated broker addresses and client.id for producer identification
3. Set acks to all (or -1) for maximum durability; 1 for leader acknowledgment; 0 for fire-and-forget
4. Choose serializer: StringSerializer for keys/values or implement Serializer<T> for custom objects
5. Set retries to handle transient broker failures and max.in.flight.requests.per.connection to balance throughput vs ordering
6. Configure partitioner: use DefaultPartitioner (hash-based) or implement Partitioner for custom logic
7. Set compression.type to snappy or lz4 to reduce network I/O for high-volume scenarios
8. Add callback handlers to send() to track delivery status and handle errors explicitly

Code

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.concurrent.Future;

public class KafkaProducerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        
        // Broker connectivity
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.CLIENT_ID_CONFIG, "my-producer");
        
        // Serialization
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        
        // Delivery guarantees
        props.put(ProducerConfig.ACKS_CONFIG, "all");  // Wait for all in-sync replicas
        props.put(ProducerConfig.RETRIES_CONFIG, 3);
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);  // Enforce ordering
        
        // Performance tuning
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);  // 16KB bat

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change