[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:08 dist-test-slave-tsm4 krb5kdc[2889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:08 dist-test-slave-tsm4 krb5kdc[2889](info): set up 2 sockets
Jun 20 19:00:08 dist-test-slave-tsm4 krb5kdc[2889](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:00:09.672209 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:09 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:09.786751 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38071
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:09.892889 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:09.893169 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:09.893255 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:09.897214 2902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:09.897285 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:09.897321 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:09.897378 2902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:09.897413 2902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:09.902359 2902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:38071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:09.903487 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:09.904458 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:09.912914 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:09.912920 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:09.912981 2902 server_base.cc:1034] running on GCE node
W20260620 19:00:09.912979 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:09.913759 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:09.914814 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:09.916006 2902 hybrid_clock.cc:648] HybridClock initialized: now 1781982009915964 us; error 62 us; skew 500 ppm
Jun 20 19:00:09 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:09.918780 2902 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:09.919762 2902 webserver.cc:460] Webserver started at http://127.2.208.190:37419/ using document root <none> and password file <none>
I20260620 19:00:09.920264 2902 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:09.920321 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:09.920485 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:09.922441 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "5826c04cef7a43a18d47b88db9166976"
format_stamp: "Formatted at 2026-06-20 19:00:09 on dist-test-slave-tsm4"
server_key: "02d8fc54c03cee7e6c54e17583e96d57"
server_key_iv: "4cd688d85d18b33f34994b2fdcd99e31"
server_key_version: "encryptionkey@0"
I20260620 19:00:09.922950 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "5826c04cef7a43a18d47b88db9166976"
format_stamp: "Formatted at 2026-06-20 19:00:09 on dist-test-slave-tsm4"
server_key: "02d8fc54c03cee7e6c54e17583e96d57"
server_key_iv: "4cd688d85d18b33f34994b2fdcd99e31"
server_key_version: "encryptionkey@0"
I20260620 19:00:09.926424 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:09.928807 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:09.929984 2902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:09.930109 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "5826c04cef7a43a18d47b88db9166976"
format_stamp: "Formatted at 2026-06-20 19:00:09 on dist-test-slave-tsm4"
server_key: "02d8fc54c03cee7e6c54e17583e96d57"
server_key_iv: "4cd688d85d18b33f34994b2fdcd99e31"
server_key_version: "encryptionkey@0"
I20260620 19:00:09.930219 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:09.946954 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:09.950170 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:09.950362 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:09.956948 2902 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:38071
I20260620 19:00:09.956970 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:38071 every 8 connection(s)
I20260620 19:00:09.957826 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:09.960295 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:09.962391 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 2902
I20260620 19:00:09.962553 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:09.964146 2882 external_mini_cluster.cc:1428] Setting key 28f2d67eea16c454467ecb5fa9c3477d
I20260620 19:00:09.966724 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: Bootstrap starting.
I20260620 19:00:09.969228 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:09.970075 2970 log.cc:824] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:09.972164 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: No bootstrap required, opened a new log
I20260620 19:00:09.976372 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } }
I20260620 19:00:09.976572 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:09.976603 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5826c04cef7a43a18d47b88db9166976, State: Initialized, Role: FOLLOWER
I20260620 19:00:09.977114 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } }
I20260620 19:00:09.977240 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:09.977316 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:09.977425 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:09.978422 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } }
I20260620 19:00:09.978739 2970 leader_election.cc:302] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5826c04cef7a43a18d47b88db9166976; no voters:
I20260620 19:00:09.979059 2970 leader_election.cc:288] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:09.979169 2975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:09.979408 2975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [term 1 LEADER]: Becoming Leader. State: Replica: 5826c04cef7a43a18d47b88db9166976, State: Running, Role: LEADER
I20260620 19:00:09.979724 2975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } }
I20260620 19:00:09.980268 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:09.981411 2976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5826c04cef7a43a18d47b88db9166976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } } }
I20260620 19:00:09.981416 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5826c04cef7a43a18d47b88db9166976. Latest consensus state: current_term: 1 leader_uuid: "5826c04cef7a43a18d47b88db9166976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5826c04cef7a43a18d47b88db9166976" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38071 } } }
I20260620 19:00:09.981557 2976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:09.981560 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:00:09 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
W20260620 19:00:09.985184 2990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:09.985260 2990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:09.985320 2984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:09.986079 2984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:09.990479 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:09.967654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.2.208.190:38071)
0620 19:00:09.968101 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:09.968120 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:09.969027 (+ 907us) server_negotiation.cc:407] Connection header received
0620 19:00:09.972828 (+ 3801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:09.972843 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:09.973110 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:09.973368 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:09.976183 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:09.977593 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:09.979443 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:09.980162 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:09.980283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:09.985175 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:09.985179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:09.985191 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:09.985213 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:09.986836 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:09.987351 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:09.987396 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:09.987404 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:09.987514 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:09.987935 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:09.987939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:09.987940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:09.988440 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:09.988740 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:09.989264 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 19:00:09.989556 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260620 19:00:09.991276 2984 catalog_manager.cc:1347] Generated new cluster ID: f5bf7802102a4739b4f7f0b0b55776c2
I20260620 19:00:09.991408 2984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:10.017053 2984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:10.018198 2984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:10.026625 2984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5826c04cef7a43a18d47b88db9166976: Generated new TSK 0
I20260620 19:00:10.027316 2984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:10.052210 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:10.158178 2996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:10.158373 2996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:10.158416 2996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:10.162355 2996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:10.162402 2996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:10.162456 2996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:10.168313 2996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:10.169723 2996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:10.171109 2996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:10.183344 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:10.183558 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:10.183625 2996 server_base.cc:1034] running on GCE node
W20260620 19:00:10.183914 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:10.184406 2996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:10.185214 2996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:10.186389 2996 hybrid_clock.cc:648] HybridClock initialized: now 1781982010186357 us; error 60 us; skew 500 ppm
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:10.190240 2996 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:10.191308 2996 webserver.cc:460] Webserver started at http://127.2.208.129:38529/ using document root <none> and password file <none>
I20260620 19:00:10.192057 2996 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:10.192157 2996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:10.192412 2996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:10.195302 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "d49b9bc9979a4a45a4c479572d206930"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "2691ddec5cf226b9064bcc7ef38d7b87"
server_key_iv: "09ee7fb02d77d026704df072a35f55b2"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.196010 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "d49b9bc9979a4a45a4c479572d206930"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "2691ddec5cf226b9064bcc7ef38d7b87"
server_key_iv: "09ee7fb02d77d026704df072a35f55b2"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.200881 2996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:00:10.204802 3011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.206323 2996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:00:10.206445 2996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "d49b9bc9979a4a45a4c479572d206930"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "2691ddec5cf226b9064bcc7ef38d7b87"
server_key_iv: "09ee7fb02d77d026704df072a35f55b2"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.206569 2996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:10.240972 2996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:10.244081 2996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:10.244233 2996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:10.244733 2996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:10.245472 2996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:10.245517 2996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.245553 2996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:10.245568 2996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.254669 2996 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:36329
I20260620 19:00:10.254702 3124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:36329 every 8 connection(s)
I20260620 19:00:10.255556 2996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:10.257680 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 2996
I20260620 19:00:10.257805 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:10.258104 2882 external_mini_cluster.cc:1428] Setting key 0cbbf7c676d80c932c61e654d9a751ad
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:10.270277 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.257201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:53973 (local address 127.2.208.190:38071)
0620 19:00:10.257345 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:10.257349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:10.258182 (+ 833us) server_negotiation.cc:407] Connection header received
0620 19:00:10.259925 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:10.259927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:10.259968 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:10.260030 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:10.261584 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.262476 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.263670 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.264074 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.264178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.266786 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:10.266792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:10.266799 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:10.266835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:10.268505 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.268948 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.268954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.268958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.269033 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.269326 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.269328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.269329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.269498 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:10.269581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:10.270064 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 19:00:10.270156 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:00:10.271189 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.257493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38071 (local address 127.2.208.129:53973)
0620 19:00:10.257994 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:10.258067 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:10.259713 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:10.260230 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:10.260241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:10.260754 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:10.261421 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.261444 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.262597 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.262599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.263531 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.263539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.264223 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.264226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.264364 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.264967 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:10.264982 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:10.266626 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:10.268602 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.268606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.268617 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.268855 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.269114 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.269117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.269118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.269249 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.269595 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:10.269599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:10.269833 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:00:10.270014 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 19:00:10.270251 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260620 19:00:10.272358 3125 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38071
I20260620 19:00:10.272590 3125 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:10.273031 3125 heartbeater.cc:511] Master 127.2.208.190:38071 requested a full tablet report, sending...
I20260620 19:00:10.274425 2934 ts_manager.cc:194] Registered new tserver with Master: d49b9bc9979a4a45a4c479572d206930 (127.2.208.129:36329)
I20260620 19:00:10.275564 2934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:53973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:10.283915 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:10.392329 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:10.392545 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:10.392586 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:10.396375 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:10.396422 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:10.396477 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:10.401444 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:10.402367 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:10.403276 3130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:10.412714 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:10.412818 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:10.412703 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:10.413539 3130 server_base.cc:1034] running on GCE node
I20260620 19:00:10.413897 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:10.414489 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:10.415654 3130 hybrid_clock.cc:648] HybridClock initialized: now 1781982010415621 us; error 43 us; skew 500 ppm
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:10.418236 3130 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:10.419147 3130 webserver.cc:460] Webserver started at http://127.2.208.130:46429/ using document root <none> and password file <none>
I20260620 19:00:10.419734 3130 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:10.419795 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:10.419955 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:10.421926 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "f5f358b7a7fd482fb85bff4edd5bfe47"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "13e00d00b59b458b07211219d0e9819c"
server_key_iv: "f82a8553b9bd78371511bc2b7fc46c01"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.422453 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "f5f358b7a7fd482fb85bff4edd5bfe47"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "13e00d00b59b458b07211219d0e9819c"
server_key_iv: "f82a8553b9bd78371511bc2b7fc46c01"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.425971 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:00:10.428258 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.429455 3130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:10.429584 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "f5f358b7a7fd482fb85bff4edd5bfe47"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "13e00d00b59b458b07211219d0e9819c"
server_key_iv: "f82a8553b9bd78371511bc2b7fc46c01"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.429693 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:10.444576 3130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:10.447208 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:10.447446 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:10.448060 3130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:10.448891 3130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:10.448956 3130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.449020 3130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:10.449066 3130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.457886 3130 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:33035
I20260620 19:00:10.457914 3258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:33035 every 8 connection(s)
I20260620 19:00:10.458709 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:10.459192 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3130
I20260620 19:00:10.459316 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:10.459586 2882 external_mini_cluster.cc:1428] Setting key 39ca272a9fb16fa12d0b3833fac3abb6
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
I20260620 19:00:10.474165 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.460459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:41777 (local address 127.2.208.190:38071)
0620 19:00:10.460580 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:10.460584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:10.461513 (+ 929us) server_negotiation.cc:407] Connection header received
0620 19:00:10.463043 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:10.463046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:10.463088 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:10.463162 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:10.464736 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.465332 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.467202 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.467474 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.467552 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.470265 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:10.470271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:10.470278 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:10.470301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:10.471838 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.472478 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.472480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.472481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.472557 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.472932 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.472934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.472935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.473106 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:10.473183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:10.473799 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 19:00:10.473939 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:00:10.474949 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.460910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38071 (local address 127.2.208.130:41777)
0620 19:00:10.461382 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:10.461422 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:10.462813 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:10.463351 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:10.463383 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:10.463906 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:10.464573 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.464594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.465463 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.465471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.467087 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.467099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.467615 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.467619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.467841 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.468444 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:10.468469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:10.470107 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:10.471957 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.471968 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.471981 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.472386 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.472641 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.472644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.472647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.472837 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.473207 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:10.473216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:10.473528 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 19:00:10.473742 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 19:00:10.474066 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260620 19:00:10.476167 3259 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38071
I20260620 19:00:10.476442 3259 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:10.477037 3259 heartbeater.cc:511] Master 127.2.208.190:38071 requested a full tablet report, sending...
I20260620 19:00:10.478178 2934 ts_manager.cc:194] Registered new tserver with Master: f5f358b7a7fd482fb85bff4edd5bfe47 (127.2.208.130:33035)
I20260620 19:00:10.478737 2934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:41777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:10.490193 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:10.599334 3264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:10.599601 3264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:10.599640 3264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:10.604326 3264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:10.604383 3264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:10.604496 3264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:10.609810 3264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:10.610738 3264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:10.611718 3264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:10.620894 3270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:10.620878 3272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:10.620882 3269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:10.621296 3264 server_base.cc:1034] running on GCE node
I20260620 19:00:10.621668 3264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:10.622259 3264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:10.623430 3264 hybrid_clock.cc:648] HybridClock initialized: now 1781982010623394 us; error 44 us; skew 500 ppm
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:10.626470 3264 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:10.627236 3264 webserver.cc:460] Webserver started at http://127.2.208.131:41271/ using document root <none> and password file <none>
I20260620 19:00:10.627790 3264 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:10.627851 3264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:10.628010 3264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:10.629904 3264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "331f992e43d642878e9cf54274938360"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "782ebd87e8e08ba30bb84370c7cd981f"
server_key_iv: "155a7c7eee33424a054645e1ddf89cf3"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.630407 3264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "331f992e43d642878e9cf54274938360"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "782ebd87e8e08ba30bb84370c7cd981f"
server_key_iv: "155a7c7eee33424a054645e1ddf89cf3"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.634093 3264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:10.636421 3279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.637552 3264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:10.637663 3264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "331f992e43d642878e9cf54274938360"
format_stamp: "Formatted at 2026-06-20 19:00:10 on dist-test-slave-tsm4"
server_key: "782ebd87e8e08ba30bb84370c7cd981f"
server_key_iv: "155a7c7eee33424a054645e1ddf89cf3"
server_key_version: "encryptionkey@0"
I20260620 19:00:10.637754 3264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:10.690090 3264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:10.693089 3264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:10.693287 3264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:10.693890 3264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:10.694720 3264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:10.694765 3264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.694826 3264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:10.694862 3264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:10.704419 3264 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:42403
I20260620 19:00:10.704473 3392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:42403 every 8 connection(s)
I20260620 19:00:10.705328 3264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:10.706447 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3264
I20260620 19:00:10.706552 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:10.706812 2882 external_mini_cluster.cc:1428] Setting key 520497adc2caa1892192695aede7b235
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:10.718995 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.707212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:48599 (local address 127.2.208.190:38071)
0620 19:00:10.707395 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:10.707400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:10.708420 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 19:00:10.709983 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:10.709985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:10.710029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:10.710093 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:10.711122 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.711747 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.712944 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.713155 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.713230 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.715577 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:10.715580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:10.715582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:10.715604 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:10.716969 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.717535 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.717538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.717540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.717618 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.717958 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.717961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.717963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.718136 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:10.718236 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:10.718703 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 19:00:10.718821 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:00:10.719836 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.707713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38071 (local address 127.2.208.131:48599)
0620 19:00:10.708249 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:10.708316 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:10.709804 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:10.710222 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:10.710229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:10.710558 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:10.711012 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.711024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.711846 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.711849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.712848 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:10.712856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.713270 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.713272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:10.713403 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.714002 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:10.714018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:10.715427 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:10.717131 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.717137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.717149 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.717407 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.717708 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:10.717712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:10.717714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:10.717850 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:10.718235 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:10.718241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:10.718469 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 19:00:10.718649 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 19:00:10.718941 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260620 19:00:10.720923 3393 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38071
I20260620 19:00:10.721151 3393 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:10.721570 3393 heartbeater.cc:511] Master 127.2.208.190:38071 requested a full tablet report, sending...
I20260620 19:00:10.722465 2934 ts_manager.cc:194] Registered new tserver with Master: 331f992e43d642878e9cf54274938360 (127.2.208.131:42403)
I20260620 19:00:10.723030 2934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:48599
I20260620 19:00:10.732834 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:10.760500 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.751446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.2.208.129:36329)
0620 19:00:10.751797 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:10.751810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:10.751836 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:00:10.751944 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:10.751950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:10.752152 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:10.752269 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:10.752876 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.753631 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.754392 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.754757 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.754838 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.757346 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:10.757354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:10.757367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:10.757388 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:10.758813 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.759203 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.759208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.759209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.759281 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.759571 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.759573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.759574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.759792 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:10.759940 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:10.760171 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 19:00:10.760346 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":159,"threads_started":1}
W20260620 19:00:10.761461 3039 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41850
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:10.786003 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:10.777757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.2.208.129:36329)
0620 19:00:10.777952 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:10.777957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:10.778086 (+ 129us) server_negotiation.cc:407] Connection header received
0620 19:00:10.778247 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:10.778250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:10.778298 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:10.778389 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:10.779107 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.779703 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.780589 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:10.780822 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:10.780900 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:10.783336 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:10.783339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:10.783341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:10.783379 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:10.784599 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.784960 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.784962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.784963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.785030 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:10.785268 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:10.785270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:10.785271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:10.785410 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:10.785500 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:10.785693 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:00:10.785779 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 19:00:10.787050 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 2996
I20260620 19:00:10.793210 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3130
I20260620 19:00:10.800225 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3264
I20260620 19:00:10.805370 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 2902
2026-06-20T19:00:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2174 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[3407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[3407](info): set up 2 sockets
Jun 20 19:00:10 dist-test-slave-tsm4 krb5kdc[3407](info): commencing operation
krb5kdc: starting...
W20260620 19:00:11.838866 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:11 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:11.958395 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37781
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:12.063992 3420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:12.064278 3420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:12.064345 3420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:12.068284 3420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:12.068372 3420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:12.068409 3420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:12.068444 3420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:12.068480 3420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:12.073508 3420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:37781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:12.074518 3420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:12.075490 3420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:12.083567 3428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.083662 3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.083577 3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:12.083653 3420 server_base.cc:1034] running on GCE node
I20260620 19:00:12.084415 3420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:12.085441 3420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:12.086658 3420 hybrid_clock.cc:648] HybridClock initialized: now 1781982012086644 us; error 35 us; skew 500 ppm
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:12.089344 3420 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:12.090145 3420 webserver.cc:460] Webserver started at http://127.2.208.190:35821/ using document root <none> and password file <none>
I20260620 19:00:12.090621 3420 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:12.090687 3420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:12.090835 3420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:12.092764 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "388277b06ec342e997ef4252c40274f5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "65eaea7350662cb3f53df0d5cc92aff5"
server_key_iv: "50e28b79b2dd3cdc88a9ee9a4763741d"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.093297 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "388277b06ec342e997ef4252c40274f5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "65eaea7350662cb3f53df0d5cc92aff5"
server_key_iv: "50e28b79b2dd3cdc88a9ee9a4763741d"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.096726 3420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:12.098862 3435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.100049 3420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:12.100196 3420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "388277b06ec342e997ef4252c40274f5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "65eaea7350662cb3f53df0d5cc92aff5"
server_key_iv: "50e28b79b2dd3cdc88a9ee9a4763741d"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.100288 3420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:12.109544 3420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:12.112144 3420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:12.112353 3420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:12.119036 3487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:37781 every 8 connection(s)
I20260620 19:00:12.119046 3420 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:37781
I20260620 19:00:12.119946 3420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:12.122480 3488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:12.123531 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3420
I20260620 19:00:12.123642 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:12.123895 2882 external_mini_cluster.cc:1428] Setting key 4fc0c0597a4c0699df17daffe6b885df
I20260620 19:00:12.128607 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: Bootstrap starting.
I20260620 19:00:12.130775 3488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:12.131518 3488 log.cc:824] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:12.133458 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: No bootstrap required, opened a new log
I20260620 19:00:12.136876 3488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } }
I20260620 19:00:12.137068 3488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:12.137140 3488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388277b06ec342e997ef4252c40274f5, State: Initialized, Role: FOLLOWER
I20260620 19:00:12.137290 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.125083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.2.208.190:37781)
0620 19:00:12.125437 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.125458 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.125490 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:00:12.126117 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.126132 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.126464 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.126723 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:12.127636 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.128703 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.129608 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.130064 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.130239 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.132884 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:12.132897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:12.132909 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:12.132934 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:12.134574 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.135004 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.135010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.135015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.135122 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.135423 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.135427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.135429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.135772 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:12.135980 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.136279 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.136528 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260620 19:00:12.137598 3488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } }
I20260620 19:00:12.137768 3488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:12.137854 3488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:12.137946 3488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:12.138731 3488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } }
I20260620 19:00:12.139052 3488 leader_election.cc:302] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 388277b06ec342e997ef4252c40274f5; no voters:
I20260620 19:00:12.139379 3488 leader_election.cc:288] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:12.139531 3493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:12.139784 3493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [term 1 LEADER]: Becoming Leader. State: Replica: 388277b06ec342e997ef4252c40274f5, State: Running, Role: LEADER
I20260620 19:00:12.140069 3493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } }
I20260620 19:00:12.140532 3488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:12.141638 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 388277b06ec342e997ef4252c40274f5. Latest consensus state: current_term: 1 leader_uuid: "388277b06ec342e997ef4252c40274f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } } }
I20260620 19:00:12.141798 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:12.142177 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "388277b06ec342e997ef4252c40274f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388277b06ec342e997ef4252c40274f5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37781 } } }
I20260620 19:00:12.142865 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5 [sys.catalog]: This master's current role is: LEADER
W20260620 19:00:12.145184 3509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:12.145247 3509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:12.145335 3502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:12.146073 3502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:12.150856 3502 catalog_manager.cc:1347] Generated new cluster ID: 9a9156ba2ddd431bbc490765f6a3f3b1
I20260620 19:00:12.150913 3502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:12.161430 3502 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:12.162544 3502 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:12.167740 3502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 388277b06ec342e997ef4252c40274f5: Generated new TSK 0
I20260620 19:00:12.168323 3502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:12.195107 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:12.303507 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:12.303807 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:12.303886 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:12.307972 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:12.308028 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:12.308085 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:12.313102 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:12.314119 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:12.315111 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:12.324383 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.324458 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.324389 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:12.324767 3514 server_base.cc:1034] running on GCE node
I20260620 19:00:12.325098 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:12.325670 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:12.326858 3514 hybrid_clock.cc:648] HybridClock initialized: now 1781982012326848 us; error 25 us; skew 500 ppm
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:12.329444 3514 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:12.330261 3514 webserver.cc:460] Webserver started at http://127.2.208.129:40797/ using document root <none> and password file <none>
I20260620 19:00:12.330792 3514 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:12.330885 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:12.331102 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:12.333047 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "c18e8f7ffb794f91b57c05dcd3c91a67"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "9f4fd61c7d635dd4b0199d729164128c"
server_key_iv: "f1bd664b1dafff866ba21de2b23d11ab"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.333658 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "c18e8f7ffb794f91b57c05dcd3c91a67"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "9f4fd61c7d635dd4b0199d729164128c"
server_key_iv: "f1bd664b1dafff866ba21de2b23d11ab"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.337108 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:12.339254 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.340404 3514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:12.340534 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "c18e8f7ffb794f91b57c05dcd3c91a67"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "9f4fd61c7d635dd4b0199d729164128c"
server_key_iv: "f1bd664b1dafff866ba21de2b23d11ab"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.340639 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:12.356541 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:12.359130 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:12.359330 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:12.359900 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:12.360728 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:12.360801 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.360869 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:12.360927 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.370009 3514 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:38371
I20260620 19:00:12.370035 3642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:38371 every 8 connection(s)
I20260620 19:00:12.370889 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:12.381047 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3514
I20260620 19:00:12.381150 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:12.381435 2882 external_mini_cluster.cc:1428] Setting key b565fc36574977fe9a33b758bb4e38a6
I20260620 19:00:12.384471 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.372620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:49829 (local address 127.2.208.190:37781)
0620 19:00:12.372762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.372767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.373595 (+ 828us) server_negotiation.cc:407] Connection header received
0620 19:00:12.374857 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.374860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.374907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.374983 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:12.376201 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.376723 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.377846 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.378069 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.378148 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.380504 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:12.380508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:12.380511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:12.380536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:12.382227 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.382797 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.382800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.382801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.382867 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.383265 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.383270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.383273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.383528 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:12.383630 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.384071 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.384209 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:00:12.385227 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.373025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37781 (local address 127.2.208.129:49829)
0620 19:00:12.373448 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.373490 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.374664 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.375149 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.375155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.375587 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:12.376054 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.376067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.376830 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.376832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.377736 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.377744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.378197 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.378199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.378344 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.378919 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:12.378934 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:12.380358 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:12.382332 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.382337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.382350 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.382666 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.383001 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.383004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.383005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.383158 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.383641 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:12.383645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:12.383845 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.383991 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.384217 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260620 19:00:12.386440 3643 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37781
I20260620 19:00:12.386665 3643 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:12.387219 3643 heartbeater.cc:511] Master 127.2.208.190:37781 requested a full tablet report, sending...
I20260620 19:00:12.388665 3452 ts_manager.cc:194] Registered new tserver with Master: c18e8f7ffb794f91b57c05dcd3c91a67 (127.2.208.129:38371)
I20260620 19:00:12.389747 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:49829
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:12.406351 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:12.514089 3648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:12.514317 3648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:12.514358 3648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:12.518307 3648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:12.518357 3648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:12.518411 3648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:12.523260 3648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:12.524156 3648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:12.525031 3648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:12.534274 3653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.534291 3654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:12.534312 3648 server_base.cc:1034] running on GCE node
W20260620 19:00:12.534329 3656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:12.534870 3648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:12.535524 3648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:12.536707 3648 hybrid_clock.cc:648] HybridClock initialized: now 1781982012536683 us; error 46 us; skew 500 ppm
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:12.539266 3648 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:12.540175 3648 webserver.cc:460] Webserver started at http://127.2.208.130:38295/ using document root <none> and password file <none>
I20260620 19:00:12.540737 3648 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:12.540832 3648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:12.541045 3648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:12.543208 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "0ffd36b21883443f9898aa35bca22b3f"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "701e9f574af52703497d3efb381fa509"
server_key_iv: "c0321b4a396298ec30b255ef04b82038"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.543880 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "0ffd36b21883443f9898aa35bca22b3f"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "701e9f574af52703497d3efb381fa509"
server_key_iv: "c0321b4a396298ec30b255ef04b82038"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.547281 3648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:12.549541 3663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.550724 3648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:12.550858 3648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "0ffd36b21883443f9898aa35bca22b3f"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "701e9f574af52703497d3efb381fa509"
server_key_iv: "c0321b4a396298ec30b255ef04b82038"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.550966 3648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:12.562369 3648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:12.564991 3648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:12.565194 3648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:12.565727 3648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:12.566498 3648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:12.566574 3648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.566638 3648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:12.566674 3648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.575295 3648 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:38907
I20260620 19:00:12.575316 3776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:38907 every 8 connection(s)
I20260620 19:00:12.576131 3648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:12.581560 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3648
I20260620 19:00:12.581647 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:12.581930 2882 external_mini_cluster.cc:1428] Setting key 5a34b57d60df0d29635714d112358f23
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:12.589871 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.577869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:33593 (local address 127.2.208.190:37781)
0620 19:00:12.578004 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.578008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.578734 (+ 726us) server_negotiation.cc:407] Connection header received
0620 19:00:12.580089 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.580092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.580148 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.580226 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:12.581280 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.581941 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.583492 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.583713 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.583830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.586334 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:12.586338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:12.586340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:12.586370 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:12.587872 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.588372 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.588378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.588379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.588459 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.588823 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.588827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.588829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.589003 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:12.589114 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.589596 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.589708 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:00:12.590615 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.578209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37781 (local address 127.2.208.130:33593)
0620 19:00:12.578595 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.578632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.579856 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.580378 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.580384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.580701 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:12.581114 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.581135 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.582100 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.582103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.583374 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.583387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.583860 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.583864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.584030 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.584592 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:12.584608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:12.586178 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:12.588009 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.588014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.588027 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.588269 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.588561 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.588564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.588566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.588721 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.589122 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:12.589129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:12.589343 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.589539 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.589838 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260620 19:00:12.591702 3777 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37781
I20260620 19:00:12.591914 3777 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:12.592361 3777 heartbeater.cc:511] Master 127.2.208.190:37781 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
I20260620 19:00:12.593451 3452 ts_manager.cc:194] Registered new tserver with Master: 0ffd36b21883443f9898aa35bca22b3f (127.2.208.130:38907)
I20260620 19:00:12.594022 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:33593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:12.608060 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:12.714969 3782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:12.715255 3782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:12.715330 3782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:12.719224 3782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:12.719311 3782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:12.719453 3782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:12.724547 3782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:12.725504 3782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:12.726467 3782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:12.735944 3790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.735951 3787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:12.736042 3788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:12.736184 3782 server_base.cc:1034] running on GCE node
I20260620 19:00:12.736646 3782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:12.737255 3782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:12.738448 3782 hybrid_clock.cc:648] HybridClock initialized: now 1781982012738428 us; error 32 us; skew 500 ppm
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:12.741151 3782 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:12.742136 3782 webserver.cc:460] Webserver started at http://127.2.208.131:33335/ using document root <none> and password file <none>
I20260620 19:00:12.742681 3782 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:12.742775 3782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:12.742972 3782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:12.745188 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "052297564e75460e9fce18ec18c183d5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "90eea3c6e7262ed84f8f04f8cf43de2c"
server_key_iv: "a4c4a14cc1effd992c7cbb8bc27f03e2"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.745749 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "052297564e75460e9fce18ec18c183d5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "90eea3c6e7262ed84f8f04f8cf43de2c"
server_key_iv: "a4c4a14cc1effd992c7cbb8bc27f03e2"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.749267 3782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 19:00:12.751528 3797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.752609 3782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:00:12.752763 3782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "052297564e75460e9fce18ec18c183d5"
format_stamp: "Formatted at 2026-06-20 19:00:12 on dist-test-slave-tsm4"
server_key: "90eea3c6e7262ed84f8f04f8cf43de2c"
server_key_iv: "a4c4a14cc1effd992c7cbb8bc27f03e2"
server_key_version: "encryptionkey@0"
I20260620 19:00:12.752871 3782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:12.775796 3782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:12.778312 3782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:12.778502 3782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:12.779031 3782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:12.779845 3782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:12.779922 3782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.779985 3782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:12.780041 3782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:12.788139 3782 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:38163
I20260620 19:00:12.788168 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:38163 every 8 connection(s)
I20260620 19:00:12.788913 3782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:12.793562 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3782
I20260620 19:00:12.793664 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:12.793905 2882 external_mini_cluster.cc:1428] Setting key bac489eccd0c04f265a52ed2e569f406
Jun 20 19:00:12 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:12.801537 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.790674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:46873 (local address 127.2.208.190:37781)
0620 19:00:12.790817 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.790820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.791584 (+ 764us) server_negotiation.cc:407] Connection header received
0620 19:00:12.792792 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.792794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.792838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.792911 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:12.794232 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.794890 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.796320 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.796520 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.796600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.798819 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:12.798821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:12.798823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:12.798841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:12.799993 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.800407 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.800410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.800411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.800476 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.800752 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.800754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.800755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.800897 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:12.800967 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.801350 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.801432 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:00:12.802348 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.790994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37781 (local address 127.2.208.131:46873)
0620 19:00:12.791450 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.791488 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.792608 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.793066 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.793071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.793461 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:12.794065 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.794083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.795024 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.795027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.796220 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.796228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.796649 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.796652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.796777 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.797315 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:12.797329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:12.798691 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:12.800090 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.800094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.800106 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.800324 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.800552 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:12.800554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:12.800556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:12.800677 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:12.800980 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:12.800984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:12.801168 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.801304 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.801505 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260620 19:00:12.803400 3911 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37781
I20260620 19:00:12.803639 3911 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:12.804112 3911 heartbeater.cc:511] Master 127.2.208.190:37781 requested a full tablet report, sending...
I20260620 19:00:12.805085 3452 ts_manager.cc:194] Registered new tserver with Master: 052297564e75460e9fce18ec18c183d5 (127.2.208.131:38163)
I20260620 19:00:12.805627 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:46873
I20260620 19:00:12.807587 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:00:12.816857 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.810662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.2.208.190:37781)
0620 19:00:12.810769 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.810773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.810917 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:00:12.810997 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.810999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.811032 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.811087 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:12.811737 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.812200 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.812929 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.813116 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.813188 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.814262 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:12.814265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:12.814267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:12.814286 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:12.815514 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.815863 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.815866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.815867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.815932 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:12.816175 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:12.816178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:12.816179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:12.816385 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:12.816480 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.816682 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.816761 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:12.820911 3452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:12.823714 3452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:12.837085 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.831856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38371 (local address 127.0.0.1:54418)
0620 19:00:12.832289 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.832302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.832434 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.833118 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.833122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.833148 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:12.833430 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.833436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.834382 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.834388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.835613 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.835624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.836550 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.836553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.836660 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.836694 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.836823 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.836934 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":105,"spinlock_wait_cycles":20992,"thread_start_us":79,"threads_started":1}
I20260620 19:00:12.837088 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.832006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.2.208.129:38371)
0620 19:00:12.832730 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.832741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.832760 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:00:12.832827 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.832831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.832985 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.833330 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:12.833519 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.834257 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.835745 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.836421 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.836549 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.836697 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.836851 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.836941 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":133,"threads_started":1}
I20260620 19:00:12.837556 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.831214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38163 (local address 127.0.0.1:39300)
0620 19:00:12.832217 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.832288 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.832467 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.833118 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.833122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.833149 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:12.833463 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.833470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.834875 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.834878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.835999 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.836008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.837166 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.837169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.837254 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.837281 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.837355 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.837423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":146,"spinlock_wait_cycles":160128,"thread_start_us":90,"threads_started":1}
I20260620 19:00:12.837960 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.831200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38907 (local address 127.0.0.1:44202)
0620 19:00:12.832217 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.832290 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.832434 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.833034 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.833040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.833061 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:12.833394 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.833407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.835445 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.835447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.836406 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.836416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.837571 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.837577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.837677 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.837711 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.837777 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.837837 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":151,"spinlock_wait_cycles":21760,"thread_start_us":59,"threads_started":1}
I20260620 19:00:12.837988 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.831287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.2.208.131:38163)
0620 19:00:12.831738 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.831749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.832425 (+ 676us) server_negotiation.cc:407] Connection header received
0620 19:00:12.832608 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.832614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.832808 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.832943 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:12.833563 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.834715 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.836131 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.836876 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.836974 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.837093 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.837769 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.837855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":164,"threads_started":1}
I20260620 19:00:12.839550 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.831573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.2.208.130:38907)
0620 19:00:12.831986 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.831997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.832575 (+ 578us) server_negotiation.cc:407] Connection header received
0620 19:00:12.832654 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.832660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.832857 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.833045 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:12.833546 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.834722 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.836540 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.837307 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.837396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.837513 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.838929 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.839006 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260620 19:00:12.840246 3577 tablet_service.cc:1467] Processing CreateTablet for tablet c799b9c81de34cf4b647747daacbaa79 (DEFAULT_TABLE table=test-table [id=f6f8e6d0e6f44586b8b481b991454a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:12.840649 3845 tablet_service.cc:1467] Processing CreateTablet for tablet c799b9c81de34cf4b647747daacbaa79 (DEFAULT_TABLE table=test-table [id=f6f8e6d0e6f44586b8b481b991454a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:12.841466 3577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799b9c81de34cf4b647747daacbaa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:12.841635 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799b9c81de34cf4b647747daacbaa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:12.848781 3928 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Bootstrap starting.
I20260620 19:00:12.848781 3929 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Bootstrap starting.
I20260620 19:00:12.851186 3928 tablet_bootstrap.cc:654] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:12.851500 3711 tablet_service.cc:1467] Processing CreateTablet for tablet c799b9c81de34cf4b647747daacbaa79 (DEFAULT_TABLE table=test-table [id=f6f8e6d0e6f44586b8b481b991454a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:12.852324 3929 tablet_bootstrap.cc:654] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:12.852468 3711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799b9c81de34cf4b647747daacbaa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:12.852681 3928 log.cc:824] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:12.853098 3929 log.cc:824] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:12.854564 3928 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: No bootstrap required, opened a new log
I20260620 19:00:12.854784 3928 ts_tablet_manager.cc:1397] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:00:12.856987 3929 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: No bootstrap required, opened a new log
I20260620 19:00:12.857169 3929 ts_tablet_manager.cc:1397] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 19:00:12.858691 3932 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Bootstrap starting.
I20260620 19:00:12.858572 3928 raft_consensus.cc:348] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.858832 3928 raft_consensus.cc:374] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:12.858891 3928 raft_consensus.cc:729] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18e8f7ffb794f91b57c05dcd3c91a67, State: Initialized, Role: FOLLOWER
I20260620 19:00:12.859354 3928 consensus_queue.cc:260] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.860450 3643 heartbeater.cc:503] Master 127.2.208.190:37781 was elected leader, sending a full tablet report...
I20260620 19:00:12.861177 3929 raft_consensus.cc:348] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.861435 3929 raft_consensus.cc:374] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:12.861491 3929 raft_consensus.cc:729] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052297564e75460e9fce18ec18c183d5, State: Initialized, Role: FOLLOWER
I20260620 19:00:12.861778 3932 tablet_bootstrap.cc:654] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:12.861788 3928 ts_tablet_manager.cc:1428] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:00:12.861925 3929 consensus_queue.cc:260] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.862493 3911 heartbeater.cc:503] Master 127.2.208.190:37781 was elected leader, sending a full tablet report...
I20260620 19:00:12.862532 3932 log.cc:824] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:12.862807 3929 ts_tablet_manager.cc:1428] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 19:00:12.864674 3932 tablet_bootstrap.cc:492] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: No bootstrap required, opened a new log
I20260620 19:00:12.864907 3932 ts_tablet_manager.cc:1397] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:00:12.867893 3932 raft_consensus.cc:348] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.868098 3932 raft_consensus.cc:374] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:12.868134 3932 raft_consensus.cc:729] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffd36b21883443f9898aa35bca22b3f, State: Initialized, Role: FOLLOWER
I20260620 19:00:12.868466 3932 consensus_queue.cc:260] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.868988 3777 heartbeater.cc:503] Master 127.2.208.190:37781 was elected leader, sending a full tablet report...
I20260620 19:00:12.869225 3932 ts_tablet_manager.cc:1428] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 19:00:12.871989 3644 tablet.cc:2369] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:12.964490 3934 raft_consensus.cc:482] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:12.964684 3934 raft_consensus.cc:504] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.965790 3934 leader_election.cc:288] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ffd36b21883443f9898aa35bca22b3f (127.2.208.130:38907), c18e8f7ffb794f91b57c05dcd3c91a67 (127.2.208.129:38371)
I20260620 19:00:12.969508 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.965986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38907 (local address 127.2.208.131:55149)
0620 19:00:12.966295 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.966310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.966420 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.966715 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.966718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.966758 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:12.966870 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.966876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.967654 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.967657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.968420 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.968428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.969050 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.969053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.969155 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.969183 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.969276 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.969343 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":22,"spinlock_wait_cycles":28800}
I20260620 19:00:12.969506 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.966127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:55149 (local address 127.2.208.130:38907)
0620 19:00:12.966265 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.966270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.966404 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:00:12.966555 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.966559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.966611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.966702 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:12.966987 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.967539 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.968550 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.968931 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.969060 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.969111 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.969291 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.969355 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:12.969710 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.966374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:53803 (local address 127.2.208.129:38371)
0620 19:00:12.966506 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:12.966510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:12.966703 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:00:12.966877 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:12.966880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:12.966931 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:12.967016 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:12.967273 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.967767 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.968756 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.969136 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.969261 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.969310 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:12.969535 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:00:12.969607 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:00:12.969726 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:12.966263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38371 (local address 127.2.208.131:53803)
0620 19:00:12.966603 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:12.966614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:12.966767 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:12.967024 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:12.967027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:12.967041 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:12.967163 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.967168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.967879 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.967883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.968633 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:12.968640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:12.969252 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:12.969255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:12.969354 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:12.969382 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:00:12.969520 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 19:00:12.969584 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":1,"spinlock_wait_cycles":6912,"thread_start_us":147,"threads_started":1}
I20260620 19:00:12.970227 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799b9c81de34cf4b647747daacbaa79" candidate_uuid: "052297564e75460e9fce18ec18c183d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffd36b21883443f9898aa35bca22b3f" is_pre_election: true
I20260620 19:00:12.970396 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799b9c81de34cf4b647747daacbaa79" candidate_uuid: "052297564e75460e9fce18ec18c183d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" is_pre_election: true
I20260620 19:00:12.970521 3731 raft_consensus.cc:2454] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052297564e75460e9fce18ec18c183d5 in term 0.
I20260620 19:00:12.970630 3597 raft_consensus.cc:2454] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052297564e75460e9fce18ec18c183d5 in term 0.
I20260620 19:00:12.970988 3798 leader_election.cc:302] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052297564e75460e9fce18ec18c183d5, 0ffd36b21883443f9898aa35bca22b3f; no voters:
I20260620 19:00:12.971237 3934 raft_consensus.cc:2781] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:12.971307 3934 raft_consensus.cc:482] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:12.971335 3934 raft_consensus.cc:3037] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:12.972354 3934 raft_consensus.cc:504] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.972671 3934 leader_election.cc:288] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [CANDIDATE]: Term 1 election: Requested vote from peers 0ffd36b21883443f9898aa35bca22b3f (127.2.208.130:38907), c18e8f7ffb794f91b57c05dcd3c91a67 (127.2.208.129:38371)
I20260620 19:00:12.972982 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799b9c81de34cf4b647747daacbaa79" candidate_uuid: "052297564e75460e9fce18ec18c183d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffd36b21883443f9898aa35bca22b3f"
I20260620 19:00:12.973062 3597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799b9c81de34cf4b647747daacbaa79" candidate_uuid: "052297564e75460e9fce18ec18c183d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67"
I20260620 19:00:12.973121 3731 raft_consensus.cc:3037] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:12.973212 3597 raft_consensus.cc:3037] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:12.974149 3731 raft_consensus.cc:2454] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052297564e75460e9fce18ec18c183d5 in term 1.
I20260620 19:00:12.974189 3597 raft_consensus.cc:2454] T c799b9c81de34cf4b647747daacbaa79 P c18e8f7ffb794f91b57c05dcd3c91a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052297564e75460e9fce18ec18c183d5 in term 1.
I20260620 19:00:12.974427 3798 leader_election.cc:302] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052297564e75460e9fce18ec18c183d5, 0ffd36b21883443f9898aa35bca22b3f; no voters:
I20260620 19:00:12.974637 3934 raft_consensus.cc:2781] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:12.974844 3934 raft_consensus.cc:686] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [term 1 LEADER]: Becoming Leader. State: Replica: 052297564e75460e9fce18ec18c183d5, State: Running, Role: LEADER
I20260620 19:00:12.975152 3934 consensus_queue.cc:237] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } }
I20260620 19:00:12.978459 3452 catalog_manager.cc:5495] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 052297564e75460e9fce18ec18c183d5 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "052297564e75460e9fce18ec18c183d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052297564e75460e9fce18ec18c183d5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ffd36b21883443f9898aa35bca22b3f" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18e8f7ffb794f91b57c05dcd3c91a67" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38371 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:13.024938 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:13.016691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.2.208.129:38371)
0620 19:00:13.016861 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:13.016865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:13.016918 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:00:13.017022 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:13.017024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:13.017067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:13.017137 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:13.017797 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.018298 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.019051 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.019247 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.019320 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:13.021743 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:13.021753 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:13.021769 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:13.021791 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:13.023286 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.023745 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.023751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.023752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.023834 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.024061 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.024063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.024065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.024306 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:13.024455 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:13.024616 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:00:13.024743 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 19:00:13.039876 3912 tablet.cc:2369] T c799b9c81de34cf4b647747daacbaa79 P 052297564e75460e9fce18ec18c183d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:13.051621 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:13.043477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.2.208.129:38371)
0620 19:00:13.043635 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:13.043640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:13.043780 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:00:13.043938 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:13.043941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:13.043992 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:13.044085 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:13.044840 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.045357 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.046231 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.046461 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.046578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:13.048881 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:13.048884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:13.048886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:13.048907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:13.050176 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.050535 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.050537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.050538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.050625 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.050841 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.050843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.050844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.050978 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:13.051066 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:13.051298 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:00:13.051450 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:13.075798 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:13.068090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.2.208.129:38371)
0620 19:00:13.068273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:13.068277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:13.068332 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:00:13.068437 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:13.068439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:13.068479 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:13.068547 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:13.069149 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.069645 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.070448 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:13.070666 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:13.070783 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:13.072998 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:13.073002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:13.073010 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:13.073030 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:13.074195 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.074600 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.074605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.074610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.074685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:13.074969 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:13.074972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:13.074974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:13.075138 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:13.075273 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:13.075521 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:00:13.075638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 19:00:13.077112 3557 tablet_service.cc:2908] Rejecting scan request for tablet c799b9c81de34cf4b647747daacbaa79: Uninitialized: clean time has not yet been initialized
W20260620 19:00:13.077293 3778 tablet.cc:2369] T c799b9c81de34cf4b647747daacbaa79 P 0ffd36b21883443f9898aa35bca22b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:13.079188 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3514
I20260620 19:00:13.085397 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3648
I20260620 19:00:13.090693 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3782
I20260620 19:00:13.095988 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3420
2026-06-20T19:00:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2290 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3952](info): set up 2 sockets
Jun 20 19:00:13 dist-test-slave-tsm4 krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260620 19:00:14.126929 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:14.244716 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39029
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:14.351928 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:14.352202 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:14.352269 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:14.356180 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:14.356263 3965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:00:14.356295 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:14.356331 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:14.356362 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:14.356397 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:14.361359 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39029
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:14.362370 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:14.363330 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:14.371485 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:14.371605 3965 server_base.cc:1034] running on GCE node
W20260620 19:00:14.371614 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:14.371539 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:14.372258 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:14.373265 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:14.374457 3965 hybrid_clock.cc:648] HybridClock initialized: now 1781982014374444 us; error 29 us; skew 500 ppm
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:14.382689 3965 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:14.383658 3965 webserver.cc:460] Webserver started at http://127.2.208.190:46421/ using document root <none> and password file <none>
I20260620 19:00:14.384238 3965 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:14.384327 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:14.384563 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:14.386564 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "cadba9a16efc4261bae2af707475459a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "52fa171c5304e76c9e11f0d2348820c4"
server_key_iv: "8de9208b528f1940b142d66dd8bfe852"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.387138 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "cadba9a16efc4261bae2af707475459a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "52fa171c5304e76c9e11f0d2348820c4"
server_key_iv: "8de9208b528f1940b142d66dd8bfe852"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.390615 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:14.393025 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:14.394155 3965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:14.394291 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "cadba9a16efc4261bae2af707475459a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "52fa171c5304e76c9e11f0d2348820c4"
server_key_iv: "8de9208b528f1940b142d66dd8bfe852"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.394395 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:14.404145 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:14.407224 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:14.407482 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:14.415053 3965 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39029
I20260620 19:00:14.415055 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39029 every 8 connection(s)
I20260620 19:00:14.416013 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:14.418612 4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:14.420274 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 3965
I20260620 19:00:14.420379 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:14.420643 2882 external_mini_cluster.cc:1428] Setting key 78d03d36792ecd46b43bdaf81ea20aee
I20260620 19:00:14.424733 4043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Bootstrap starting.
I20260620 19:00:14.427155 4043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:14.428010 4043 log.cc:824] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:14.430033 4043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: No bootstrap required, opened a new log
I20260620 19:00:14.433202 4043 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } }
I20260620 19:00:14.433533 4043 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:14.433615 4043 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadba9a16efc4261bae2af707475459a, State: Initialized, Role: FOLLOWER
I20260620 19:00:14.434072 4043 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } }
I20260620 19:00:14.434224 4043 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:14.434288 4043 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:14.434293 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.421885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.2.208.190:39029)
0620 19:00:14.422240 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:14.422261 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:14.422296 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:00:14.422899 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:14.422914 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:14.423226 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:14.423564 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:14.424368 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.425407 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.426556 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.426976 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.427092 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.429680 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:14.429686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:14.429701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:14.429733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:14.431524 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.431963 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.431968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.431974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.432090 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.432373 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.432378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.432379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.432740 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:14.432962 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:14.433190 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 19:00:14.433446 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260620 19:00:14.434401 4043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:14.435251 4043 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } }
I20260620 19:00:14.435617 4043 leader_election.cc:302] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadba9a16efc4261bae2af707475459a; no voters:
I20260620 19:00:14.435913 4043 leader_election.cc:288] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:14.436043 4048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:14.436260 4048 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [term 1 LEADER]: Becoming Leader. State: Replica: cadba9a16efc4261bae2af707475459a, State: Running, Role: LEADER
I20260620 19:00:14.436559 4048 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } }
I20260620 19:00:14.437023 4043 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:14.437529 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadba9a16efc4261bae2af707475459a. Latest consensus state: current_term: 1 leader_uuid: "cadba9a16efc4261bae2af707475459a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } } }
I20260620 19:00:14.437732 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:14.437901 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadba9a16efc4261bae2af707475459a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadba9a16efc4261bae2af707475459a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39029 } } }
I20260620 19:00:14.437987 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:14.438243 4056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:14.441196 4056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:14.446270 4056 catalog_manager.cc:1347] Generated new cluster ID: 5eb1c4f0517e4601873fdbd6b17189e8
I20260620 19:00:14.446332 4056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:14.456303 4056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:14.457419 4056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:14.461637 4056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Generated new TSK 0
I20260620 19:00:14.462311 4056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:14.494324 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39029
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:14.621161 4069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:14.621402 4069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:14.621446 4069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:14.625471 4069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:14.625521 4069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:14.625559 4069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:14.625604 4069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:14.631124 4069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:14.632193 4069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:14.633129 4069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:00:14.642697 4069 server_base.cc:1034] running on GCE node
W20260620 19:00:14.642604 4075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:14.642604 4074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:14.642619 4077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:14.643435 4069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:14.644054 4069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:14.645219 4069 hybrid_clock.cc:648] HybridClock initialized: now 1781982014645205 us; error 27 us; skew 500 ppm
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:14.648176 4069 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:14.648983 4069 webserver.cc:460] Webserver started at http://127.2.208.129:35135/ using document root <none> and password file <none>
I20260620 19:00:14.649567 4069 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:14.649664 4069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:14.649896 4069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:14.652005 4069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "d3a1255df743490099d44aa2ebe98fc7"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "97f1b0be87d4a753de95c555fdc50745"
server_key_iv: "bdc1ef47054ac517ecbbe219959e7aca"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.652591 4069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a1255df743490099d44aa2ebe98fc7"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "97f1b0be87d4a753de95c555fdc50745"
server_key_iv: "bdc1ef47054ac517ecbbe219959e7aca"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.656073 4069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:14.658376 4084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:14.659735 4069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:14.659852 4069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "d3a1255df743490099d44aa2ebe98fc7"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "97f1b0be87d4a753de95c555fdc50745"
server_key_iv: "bdc1ef47054ac517ecbbe219959e7aca"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.659950 4069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:14.680234 4069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:14.683033 4069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:14.683243 4069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:14.684803 4069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:14.684877 4069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:14.684947 4069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:14.685020 4069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:14.695832 4069 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:35555
I20260620 19:00:14.696269 4199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:35555 every 8 connection(s)
I20260620 19:00:14.696913 4069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:14.699321 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.685511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37381 (local address 127.2.208.190:39029)
0620 19:00:14.685644 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:14.685648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:14.687280 (+ 1632us) server_negotiation.cc:407] Connection header received
0620 19:00:14.688529 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:14.688536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:14.688605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:14.688694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:14.690160 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.691053 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.692338 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.692731 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.692840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.695276 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:14.695280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:14.695284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:14.695318 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:14.697045 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.697684 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.697687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.697689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.697784 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.698198 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.698202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.698204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.698423 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:14.698551 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:14.699094 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 19:00:14.699202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:00:14.700093 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.686572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.129:37381)
0620 19:00:14.687132 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:14.687174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:14.688165 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:14.688859 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:14.688865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:14.689331 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:14.689977 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.689995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.691198 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.691202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.692236 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.692245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.692884 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.692887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.693029 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.693611 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:14.693626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:14.695101 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:14.697188 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.697194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.697207 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.697560 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.697913 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.697917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.697921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.698091 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.698551 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:14.698558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:14.698798 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 19:00:14.699011 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 19:00:14.699273 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":160,"threads_started":1}
I20260620 19:00:14.700605 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4069
I20260620 19:00:14.700726 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:14.701009 2882 external_mini_cluster.cc:1428] Setting key bddb9a94adfe8d79f4bfef7fd7ef2d6f
I20260620 19:00:14.701364 4200 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39029
I20260620 19:00:14.701582 4200 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:14.702075 4200 heartbeater.cc:511] Master 127.2.208.190:39029 requested a full tablet report, sending...
I20260620 19:00:14.704517 3997 ts_manager.cc:194] Registered new tserver with Master: d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555)
I20260620 19:00:14.705785 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:37381
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:00:14.714113 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.706359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:43393 (local address 127.2.208.190:39029)
0620 19:00:14.706481 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:14.706484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:14.706742 (+ 258us) server_negotiation.cc:407] Connection header received
0620 19:00:14.706844 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:14.706847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:14.706898 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:14.706982 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:14.707528 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.708411 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.709243 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.709667 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.709791 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.711045 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:14.711050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:14.711052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:14.711088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:14.712525 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.712916 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.712922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.712924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.713007 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.713302 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.713305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.713307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.713474 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:14.713587 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:14.713846 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:00:14.713948 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:00:14.714125 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.704009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.129:43393)
0620 19:00:14.706616 (+ 2607us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:14.706628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:14.706754 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:14.707021 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:14.707024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:14.707230 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:14.707337 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.707344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.708545 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.708547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.709097 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.709105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.709816 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.709822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.709932 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.710516 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:14.710542 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:14.710910 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:14.712621 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.712623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.712624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.712803 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.713112 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.713117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.713119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.713198 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.713575 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:14.713578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:14.713669 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 19:00:14.713794 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:00:14.713929 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2538,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:14.730162 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39029
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:14.839789 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:14.840026 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:14.840068 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:14.843990 4209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:14.844043 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:14.844079 4209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:14.844148 4209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:14.849215 4209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:14.850219 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:14.851212 4209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:14.860690 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:14.860739 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:14.860690 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:14.860878 4209 server_base.cc:1034] running on GCE node
I20260620 19:00:14.861375 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:14.861931 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:14.863123 4209 hybrid_clock.cc:648] HybridClock initialized: now 1781982014863110 us; error 29 us; skew 500 ppm
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:14.865978 4209 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:14.866801 4209 webserver.cc:460] Webserver started at http://127.2.208.130:45029/ using document root <none> and password file <none>
I20260620 19:00:14.867313 4209 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:14.867445 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:14.867628 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:14.869568 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "fa9ed448f38143ec82eca1a161aab24a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "43ea72954892661263a3c97b2deece1d"
server_key_iv: "ff57e93510475a4cc76bfe89605c1873"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.870070 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "fa9ed448f38143ec82eca1a161aab24a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "43ea72954892661263a3c97b2deece1d"
server_key_iv: "ff57e93510475a4cc76bfe89605c1873"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.874159 4209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:00:14.876650 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:14.877928 4209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:14.878022 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "fa9ed448f38143ec82eca1a161aab24a"
format_stamp: "Formatted at 2026-06-20 19:00:14 on dist-test-slave-tsm4"
server_key: "43ea72954892661263a3c97b2deece1d"
server_key_iv: "ff57e93510475a4cc76bfe89605c1873"
server_key_version: "encryptionkey@0"
I20260620 19:00:14.878103 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:14.894829 4209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:14.898029 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:14.898172 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:14.899633 4209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:14.899680 4209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:14.899718 4209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:14.899775 4209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:14 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:14.910337 4209 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:34535
I20260620 19:00:14.910393 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:34535 every 8 connection(s)
I20260620 19:00:14.911342 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:14.913240 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.900390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:48565 (local address 127.2.208.190:39029)
0620 19:00:14.900536 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:14.900541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:14.901383 (+ 842us) server_negotiation.cc:407] Connection header received
0620 19:00:14.902456 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:14.902461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:14.902522 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:14.902583 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:14.904103 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.904987 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.906585 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.906844 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.906964 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.909638 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:14.909642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:14.909644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:14.909669 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:14.911172 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.911762 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.911766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.911768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.911859 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.912256 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.912259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.912261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.912463 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:14.912541 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:14.913021 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 19:00:14.913122 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:00:14.914047 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.900800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.130:48565)
0620 19:00:14.901235 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:14.901281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:14.902199 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:14.903010 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:14.903018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:14.903502 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:14.903951 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.903962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.905137 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.905140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.906456 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.906465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.907007 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.907011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.907183 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.907731 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:14.907745 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:14.909461 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:14.911287 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.911295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.911311 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.911649 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.911971 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.911974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.911976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.912141 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.912558 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:14.912563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:14.912828 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 19:00:14.912969 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 19:00:14.913202 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":130,"threads_started":1}
I20260620 19:00:14.915202 4340 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39029
I20260620 19:00:14.915326 4340 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:14.915866 4340 heartbeater.cc:511] Master 127.2.208.190:39029 requested a full tablet report, sending...
I20260620 19:00:14.916219 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4209
I20260620 19:00:14.916326 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:14.916616 2882 external_mini_cluster.cc:1428] Setting key 69c058bf62b84c384989e35107c4e437
I20260620 19:00:14.917861 3997 ts_manager.cc:194] Registered new tserver with Master: fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535)
I20260620 19:00:14.918645 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:48565
I20260620 19:00:14.925606 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.917459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:33291 (local address 127.2.208.190:39029)
0620 19:00:14.917583 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:14.917588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:14.917849 (+ 261us) server_negotiation.cc:407] Connection header received
0620 19:00:14.918020 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:14.918023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:14.918058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:14.918104 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:14.918690 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.919544 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.920856 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.921092 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.921174 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.922488 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:14.922492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:14.922495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:14.922522 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:14.923981 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.924444 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.924447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.924448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.924523 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:14.924809 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:14.924811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:14.924812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:14.925001 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:14.925104 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:14.925397 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 19:00:14.925494 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:00:14.925724 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:14.917417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.130:33291)
0620 19:00:14.917774 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:14.917787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:14.917913 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:14.918236 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:14.918239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:14.918443 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:14.918558 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.918564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.919708 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.919713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.920714 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:14.920726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:14.921216 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:14.921223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:14.921357 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:14.921834 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:14.921853 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:14.922323 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:14.924096 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.924099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.924101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.924337 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.924619 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:14.924622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:14.924624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:14.924721 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:14.925129 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:14.925133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:14.925233 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 19:00:14.925351 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:00:14.925493 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:14.942595 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39029
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:15.048702 4349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:15.048977 4349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:15.049058 4349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:15.052971 4349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:15.053063 4349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:15.053135 4349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:15.053220 4349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:15.058161 4349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:15.059332 4349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:15.060601 4349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:15.070178 4355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:15.070185 4354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:15.070251 4357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:15.070362 4349 server_base.cc:1034] running on GCE node
I20260620 19:00:15.070897 4349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:15.071538 4349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:15.072726 4349 hybrid_clock.cc:648] HybridClock initialized: now 1781982015072708 us; error 32 us; skew 500 ppm
Jun 20 19:00:15 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982015, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:15.075325 4349 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:15.076140 4349 webserver.cc:460] Webserver started at http://127.2.208.131:32883/ using document root <none> and password file <none>
I20260620 19:00:15.076683 4349 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:15.076774 4349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:15.076989 4349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:15.078881 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "2217ca570cf34d2ea5eb32ef66dccd71"
format_stamp: "Formatted at 2026-06-20 19:00:15 on dist-test-slave-tsm4"
server_key: "a725a3f86beb24988e116cd690ad41e6"
server_key_iv: "67e5f5fe54387939e989ceb05d0f74c5"
server_key_version: "encryptionkey@0"
I20260620 19:00:15.079463 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "2217ca570cf34d2ea5eb32ef66dccd71"
format_stamp: "Formatted at 2026-06-20 19:00:15 on dist-test-slave-tsm4"
server_key: "a725a3f86beb24988e116cd690ad41e6"
server_key_iv: "67e5f5fe54387939e989ceb05d0f74c5"
server_key_version: "encryptionkey@0"
I20260620 19:00:15.082911 4349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:15.085222 4364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:15.086329 4349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:00:15.086457 4349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "2217ca570cf34d2ea5eb32ef66dccd71"
format_stamp: "Formatted at 2026-06-20 19:00:15 on dist-test-slave-tsm4"
server_key: "a725a3f86beb24988e116cd690ad41e6"
server_key_iv: "67e5f5fe54387939e989ceb05d0f74c5"
server_key_version: "encryptionkey@0"
I20260620 19:00:15.086563 4349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:15.118490 4349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:15.121346 4349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:15.121568 4349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:15.123018 4349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:15.123095 4349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:15.123163 4349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:15.123231 4349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 19:00:15 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982015, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:15.133848 4349 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:44381
I20260620 19:00:15.134171 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:44381 every 8 connection(s)
I20260620 19:00:15.134992 4349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:15.136883 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.123732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39323 (local address 127.2.208.190:39029)
0620 19:00:15.123862 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.123866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.125450 (+ 1584us) server_negotiation.cc:407] Connection header received
0620 19:00:15.126652 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.126655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.126713 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.126789 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:15.128131 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.128718 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.129976 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.130258 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.130381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.133027 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:15.133030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:15.133033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:15.133058 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:15.134822 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.135418 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.135425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.135430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.135520 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.135948 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.135950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.135951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.136120 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:15.136197 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.136619 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.136712 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:15.137701 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.124786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.131:39323)
0620 19:00:15.125296 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.125338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.126452 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.126954 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.126963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.127449 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:15.127968 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.127980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.128848 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.128852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.129852 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.129861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.130431 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.130436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.130592 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.131143 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:15.131158 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:15.132850 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:15.134932 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.134936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.134946 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.135271 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.135681 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.135686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.135689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.135851 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.136209 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:15.136214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:15.136428 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.136570 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.136788 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260620 19:00:15.138532 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4349
I20260620 19:00:15.138648 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:15.138922 2882 external_mini_cluster.cc:1428] Setting key 8d0f89d241c10eb2a43b46fcba876bcc
I20260620 19:00:15.139127 4480 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39029
I20260620 19:00:15.139256 4480 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:15.139845 4480 heartbeater.cc:511] Master 127.2.208.190:39029 requested a full tablet report, sending...
I20260620 19:00:15.141456 3997 ts_manager.cc:194] Registered new tserver with Master: 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131:44381)
I20260620 19:00:15.142194 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:39323
I20260620 19:00:15.148113 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.141199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:48705 (local address 127.2.208.190:39029)
0620 19:00:15.141309 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.141313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.141497 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:00:15.141581 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.141584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.141618 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.141678 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:15.142099 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.142787 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.143730 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.143959 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.144066 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.145123 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:15.145126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:15.145128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:15.145149 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:15.146483 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.146900 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.146905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.146907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.146982 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.147282 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.147285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.147287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.147522 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:15.147619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.147873 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.147966 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260620 19:00:15.148124 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.141138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.2.208.131:48705)
0620 19:00:15.141389 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.141404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.141498 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.141721 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.141726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.141908 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:15.141994 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.141999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.142909 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.142912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.143613 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.143621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.144079 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.144082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.144179 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.144680 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:15.144690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:15.144971 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:15.146614 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.146617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.146619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.146801 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.147081 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.147084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.147086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.147166 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.147620 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:15.147623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:15.147710 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.147812 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.147933 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260620 19:00:15.152381 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:00:15 dist-test-slave-tsm4 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:00:15 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:15.176067 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.168499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.2.208.190:39029)
0620 19:00:15.168626 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.168629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.168641 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:15.168675 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.168677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.168712 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.168776 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:15.169478 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.169976 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.170732 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.170927 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.171002 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.173214 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:15.173217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:15.173219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:15.173241 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:15.174619 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.174960 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.174962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.174963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.175031 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.175244 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.175247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.175248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.175461 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:15.175569 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.175765 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.175847 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:00:15.178628 3997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:15.181663 3997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:15.195408 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.190327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44381 (local address 127.0.0.1:38742)
0620 19:00:15.190618 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.190630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.190711 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.191676 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.191679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.191695 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.191766 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.191772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.193072 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.193076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.194134 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.194142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.194858 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.194956 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.194991 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.195140 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.195250 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260620 19:00:15.195417 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.190833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.2.208.131:44381)
0620 19:00:15.191221 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.191232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.191251 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:00:15.191316 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.191322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.191536 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.191902 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.192012 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.192965 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194248 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.194737 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.195013 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.195170 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.195254 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":115,"threads_started":1}
I20260620 19:00:15.195664 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.189747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:35555 (local address 127.0.0.1:50562)
0620 19:00:15.190100 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.190140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.190274 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.191421 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.191424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.191439 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.191542 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.191548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.192487 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.192493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.193851 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.193861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.195075 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.195078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.195266 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.195344 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.195432 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.195533 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":91,"spinlock_wait_cycles":9344,"thread_start_us":94,"threads_started":1}
I20260620 19:00:15.195752 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.189759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.2.208.130:34535)
0620 19:00:15.190549 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.190561 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.190613 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:00:15.190688 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.190693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.190886 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.191035 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.191509 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.192660 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194032 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.194763 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.195012 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.195534 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.195615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":95,"threads_started":1}
I20260620 19:00:15.196194 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.190522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.2.208.129:35555)
0620 19:00:15.190940 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.190955 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.190977 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:00:15.191061 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.191066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.191252 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.191416 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.191664 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.192351 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.193990 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.194763 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.194864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.195012 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.195943 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.196045 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":96,"threads_started":1}
I20260620 19:00:15.195863 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.189644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:34535 (local address 127.0.0.1:54078)
0620 19:00:15.190097 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.190140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.190273 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.191197 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.191202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.191224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.191348 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.191354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.192780 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.192784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.193907 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.193916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.195075 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.195078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.195286 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.195352 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.195432 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.195738 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":339,"spinlock_wait_cycles":10368,"thread_start_us":86,"threads_started":1}
I20260620 19:00:15.198838 4414 tablet_service.cc:1467] Processing CreateTablet for tablet c9cc064f623645b29f82baa67c0cff1d (DEFAULT_TABLE table=test-table [id=a6995b69652a46629c5de117a28ee840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.198838 4134 tablet_service.cc:1467] Processing CreateTablet for tablet c9cc064f623645b29f82baa67c0cff1d (DEFAULT_TABLE table=test-table [id=a6995b69652a46629c5de117a28ee840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.199043 4274 tablet_service.cc:1467] Processing CreateTablet for tablet c9cc064f623645b29f82baa67c0cff1d (DEFAULT_TABLE table=test-table [id=a6995b69652a46629c5de117a28ee840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.199883 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9cc064f623645b29f82baa67c0cff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.199910 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9cc064f623645b29f82baa67c0cff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.199905 4134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9cc064f623645b29f82baa67c0cff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.206379 4503 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Bootstrap starting.
I20260620 19:00:15.207350 4505 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Bootstrap starting.
I20260620 19:00:15.207535 4504 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Bootstrap starting.
I20260620 19:00:15.208444 4503 tablet_bootstrap.cc:654] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.209143 4503 log.cc:824] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:15.209429 4505 tablet_bootstrap.cc:654] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.209599 4504 tablet_bootstrap.cc:654] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.210268 4505 log.cc:824] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:15.210268 4504 log.cc:824] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:15.211103 4503 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: No bootstrap required, opened a new log
I20260620 19:00:15.211274 4503 ts_tablet_manager.cc:1397] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:15.212188 4505 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: No bootstrap required, opened a new log
I20260620 19:00:15.212240 4504 tablet_bootstrap.cc:492] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: No bootstrap required, opened a new log
I20260620 19:00:15.212335 4505 ts_tablet_manager.cc:1397] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 19:00:15.212358 4504 ts_tablet_manager.cc:1397] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:15.214982 4504 raft_consensus.cc:348] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.215004 4503 raft_consensus.cc:348] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.215214 4504 raft_consensus.cc:374] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.215222 4503 raft_consensus.cc:374] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.215265 4504 raft_consensus.cc:729] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a1255df743490099d44aa2ebe98fc7, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.215267 4503 raft_consensus.cc:729] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9ed448f38143ec82eca1a161aab24a, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.215701 4503 consensus_queue.cc:260] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.215698 4504 consensus_queue.cc:260] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.216231 4200 heartbeater.cc:503] Master 127.2.208.190:39029 was elected leader, sending a full tablet report...
I20260620 19:00:15.216136 4505 raft_consensus.cc:348] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.216325 4505 raft_consensus.cc:374] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.216365 4505 raft_consensus.cc:729] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2217ca570cf34d2ea5eb32ef66dccd71, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.216602 4504 ts_tablet_manager.cc:1428] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:00:15.216614 4503 ts_tablet_manager.cc:1428] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:00:15.216715 4505 consensus_queue.cc:260] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.217355 4340 heartbeater.cc:503] Master 127.2.208.190:39029 was elected leader, sending a full tablet report...
I20260620 19:00:15.217538 4505 ts_tablet_manager.cc:1428] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:15.217599 4480 heartbeater.cc:503] Master 127.2.208.190:39029 was elected leader, sending a full tablet report...
W20260620 19:00:15.386004 4481 tablet.cc:2369] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:15.412679 4341 tablet.cc:2369] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:15.448351 4201 tablet.cc:2369] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:15.491019 4509 raft_consensus.cc:482] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:15.491241 4509 raft_consensus.cc:504] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.492455 4509 leader_election.cc:288] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535), 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131:44381)
I20260620 19:00:15.497792 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.493193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:57249 (local address 127.2.208.131:44381)
0620 19:00:15.493351 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.493357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.493540 (+ 183us) server_negotiation.cc:407] Connection header received
0620 19:00:15.493647 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.493649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.493685 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.493782 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.493999 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.494720 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.496524 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.497149 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.497263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.497317 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.497622 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.497689 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 19:00:15.497876 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.493084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44381 (local address 127.2.208.129:57249)
0620 19:00:15.493434 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.493447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.493556 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.493802 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.493804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.493820 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.493899 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.493903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.494834 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.494837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.496393 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.496403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.497269 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.497273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.497437 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.497510 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.497579 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.497696 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":194,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260620 19:00:15.497879 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.492651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:34535 (local address 127.2.208.129:49997)
0620 19:00:15.492988 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.493007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.493149 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.493438 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.493443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.493470 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.493640 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.493646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.494687 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.494690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.496396 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.496403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.497269 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.497273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.497441 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.497494 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.497579 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.497696 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":313,"spinlock_wait_cycles":29824,"thread_start_us":105,"threads_started":1}
I20260620 19:00:15.498211 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.492787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:49997 (local address 127.2.208.130:34535)
0620 19:00:15.492947 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.492952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.493093 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:00:15.493270 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.493273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.493322 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.493400 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.493755 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.494554 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.496524 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.497139 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.497263 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.497327 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.498032 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.498107 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:00:15.498808 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9cc064f623645b29f82baa67c0cff1d" candidate_uuid: "d3a1255df743490099d44aa2ebe98fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a" is_pre_election: true
I20260620 19:00:15.498842 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9cc064f623645b29f82baa67c0cff1d" candidate_uuid: "d3a1255df743490099d44aa2ebe98fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" is_pre_election: true
I20260620 19:00:15.499099 4294 raft_consensus.cc:2454] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a1255df743490099d44aa2ebe98fc7 in term 0.
I20260620 19:00:15.499106 4434 raft_consensus.cc:2454] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a1255df743490099d44aa2ebe98fc7 in term 0.
I20260620 19:00:15.499577 4087 leader_election.cc:302] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2217ca570cf34d2ea5eb32ef66dccd71, d3a1255df743490099d44aa2ebe98fc7; no voters:
I20260620 19:00:15.499799 4509 raft_consensus.cc:2781] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:15.499900 4509 raft_consensus.cc:482] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:15.499980 4509 raft_consensus.cc:3037] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.500978 4509 raft_consensus.cc:504] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.501312 4509 leader_election.cc:288] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [CANDIDATE]: Term 1 election: Requested vote from peers fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535), 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131:44381)
I20260620 19:00:15.501665 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9cc064f623645b29f82baa67c0cff1d" candidate_uuid: "d3a1255df743490099d44aa2ebe98fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a"
I20260620 19:00:15.501766 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9cc064f623645b29f82baa67c0cff1d" candidate_uuid: "d3a1255df743490099d44aa2ebe98fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2217ca570cf34d2ea5eb32ef66dccd71"
I20260620 19:00:15.501806 4294 raft_consensus.cc:3037] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.501873 4434 raft_consensus.cc:3037] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.502840 4294 raft_consensus.cc:2454] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a1255df743490099d44aa2ebe98fc7 in term 1.
I20260620 19:00:15.503158 4087 leader_election.cc:302] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3a1255df743490099d44aa2ebe98fc7, fa9ed448f38143ec82eca1a161aab24a; no voters:
I20260620 19:00:15.503319 4509 raft_consensus.cc:2781] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:15.503352 4434 raft_consensus.cc:2454] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a1255df743490099d44aa2ebe98fc7 in term 1.
I20260620 19:00:15.503556 4509 raft_consensus.cc:686] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 1 LEADER]: Becoming Leader. State: Replica: d3a1255df743490099d44aa2ebe98fc7, State: Running, Role: LEADER
I20260620 19:00:15.503913 4509 consensus_queue.cc:237] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.506857 3995 catalog_manager.cc:5495] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129). New cstate: current_term: 1 leader_uuid: "d3a1255df743490099d44aa2ebe98fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:15.561688 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.558362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.2.208.129:35555)
0620 19:00:15.558503 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.558506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.558519 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:15.558684 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.558686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.558740 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.558819 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:15.559043 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.559547 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.560361 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.560562 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.560641 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.560914 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:15.561294 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:15.561375 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.561449 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.561495 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:00:15.567469 4434 raft_consensus.cc:1261] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Refusing update from remote peer d3a1255df743490099d44aa2ebe98fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:15.567557 4294 raft_consensus.cc:1261] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Refusing update from remote peer d3a1255df743490099d44aa2ebe98fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:15.568204 4514 consensus_queue.cc:1035] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:15.568375 4509 consensus_queue.cc:1035] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:15.575832 4519 mvcc.cc:205] Tried to move back new op lower bound from 7298998335757234176 to 7298998335506288640. Current Snapshot: MvccSnapshot[applied={T|T < 7298998335757234176}]
I20260620 19:00:15.586753 3995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:15.586935 3995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:15.589433 3995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Sending DeleteTablet for 3 replicas of tablet c9cc064f623645b29f82baa67c0cff1d
I20260620 19:00:15.590211 4134 tablet_service.cc:1514] Processing DeleteTablet for tablet c9cc064f623645b29f82baa67c0cff1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50562
I20260620 19:00:15.590358 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet c9cc064f623645b29f82baa67c0cff1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:38742
I20260620 19:00:15.590678 4274 tablet_service.cc:1514] Processing DeleteTablet for tablet c9cc064f623645b29f82baa67c0cff1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:54078
I20260620 19:00:15.591089 4528 tablet_replica.cc:330] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: stopping tablet replica
I20260620 19:00:15.591100 4526 tablet_replica.cc:330] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: stopping tablet replica
I20260620 19:00:15.591425 4526 raft_consensus.cc:2229] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:15.591391 3995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:00:15.591461 4528 raft_consensus.cc:2229] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:00:15.591770 3995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:15.591794 4526 raft_consensus.cc:2258] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:15.592065 4528 raft_consensus.cc:2258] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:15.592656 4527 tablet_replica.cc:330] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: stopping tablet replica
I20260620 19:00:15.592890 4527 raft_consensus.cc:2229] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:15.593143 4527 raft_consensus.cc:2258] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:15.593217 4526 ts_tablet_manager.cc:1905] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:15.594846 4527 ts_tablet_manager.cc:1905] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:15.596316 4526 ts_tablet_manager.cc:1918] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:15.596408 4526 log.cc:1196] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/c9cc064f623645b29f82baa67c0cff1d
I20260620 19:00:15.596735 4526 ts_tablet_manager.cc:1939] T c9cc064f623645b29f82baa67c0cff1d P d3a1255df743490099d44aa2ebe98fc7: Deleting consensus metadata
I20260620 19:00:15.597419 4527 ts_tablet_manager.cc:1918] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:15.597502 4527 log.cc:1196] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/c9cc064f623645b29f82baa67c0cff1d
I20260620 19:00:15.597815 4527 ts_tablet_manager.cc:1939] T c9cc064f623645b29f82baa67c0cff1d P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting consensus metadata
I20260620 19:00:15.597940 3982 catalog_manager.cc:4841] TS d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555): tablet c9cc064f623645b29f82baa67c0cff1d (table test-table [id=a6995b69652a46629c5de117a28ee840]) successfully deleted
I20260620 19:00:15.598474 4134 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 (DEFAULT_TABLE table=test-table [id=4d528e0c2a90491b850c791fc8c9b172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.598718 4134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5d42adcfa4498999a72f727cb6cdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.598765 4274 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 (DEFAULT_TABLE table=test-table [id=4d528e0c2a90491b850c791fc8c9b172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.598853 3983 catalog_manager.cc:4841] TS 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131:44381): tablet c9cc064f623645b29f82baa67c0cff1d (table test-table [id=a6995b69652a46629c5de117a28ee840]) successfully deleted
I20260620 19:00:15.599035 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5d42adcfa4498999a72f727cb6cdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.600752 4504 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Bootstrap starting.
I20260620 19:00:15.601280 4503 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Bootstrap starting.
I20260620 19:00:15.601429 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 (DEFAULT_TABLE table=test-table [id=4d528e0c2a90491b850c791fc8c9b172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:15.601642 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5d42adcfa4498999a72f727cb6cdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.601666 4504 tablet_bootstrap.cc:654] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.602281 4503 tablet_bootstrap.cc:654] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.603412 4503 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: No bootstrap required, opened a new log
I20260620 19:00:15.603493 4503 ts_tablet_manager.cc:1397] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:15.603770 4505 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Bootstrap starting.
I20260620 19:00:15.604249 4503 raft_consensus.cc:348] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.604354 4503 raft_consensus.cc:374] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.604385 4503 raft_consensus.cc:729] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9ed448f38143ec82eca1a161aab24a, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.604548 4504 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: No bootstrap required, opened a new log
I20260620 19:00:15.604518 4503 consensus_queue.cc:260] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.604625 4504 ts_tablet_manager.cc:1397] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:00:15.604780 4503 ts_tablet_manager.cc:1428] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:15.604861 4505 tablet_bootstrap.cc:654] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.605041 4504 raft_consensus.cc:348] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.605175 4504 raft_consensus.cc:374] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.605212 4504 raft_consensus.cc:729] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a1255df743490099d44aa2ebe98fc7, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.605319 4504 consensus_queue.cc:260] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.605587 4504 ts_tablet_manager.cc:1428] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:15.606272 4505 tablet_bootstrap.cc:492] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: No bootstrap required, opened a new log
I20260620 19:00:15.606344 4505 ts_tablet_manager.cc:1397] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:00:15.606719 4505 raft_consensus.cc:348] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.606818 4505 raft_consensus.cc:374] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.606848 4505 raft_consensus.cc:729] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2217ca570cf34d2ea5eb32ef66dccd71, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.606950 4505 consensus_queue.cc:260] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.607175 4505 ts_tablet_manager.cc:1428] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:15.608750 4528 ts_tablet_manager.cc:1905] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:15.610622 4528 ts_tablet_manager.cc:1918] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:15.610697 4528 log.cc:1196] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/c9cc064f623645b29f82baa67c0cff1d
I20260620 19:00:15.610962 4528 ts_tablet_manager.cc:1939] T c9cc064f623645b29f82baa67c0cff1d P fa9ed448f38143ec82eca1a161aab24a: Deleting consensus metadata
I20260620 19:00:15.612166 3983 catalog_manager.cc:4841] TS fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535): tablet c9cc064f623645b29f82baa67c0cff1d (table test-table [id=a6995b69652a46629c5de117a28ee840]) successfully deleted
I20260620 19:00:15.848214 4511 raft_consensus.cc:482] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:15.848352 4511 raft_consensus.cc:504] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.849526 4511 leader_election.cc:288] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555), fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535)
I20260620 19:00:15.853344 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.849788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:57597 (local address 127.2.208.129:35555)
0620 19:00:15.849921 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.849926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.850053 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:00:15.850200 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.850203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.850254 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.850355 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.850704 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.851223 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852306 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.852716 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852853 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.852912 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.853150 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.853216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:00:15.853354 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.849968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:35003 (local address 127.2.208.130:34535)
0620 19:00:15.850105 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.850108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.850123 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:15.850272 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.850278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.850325 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.850420 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:15.850752 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.851233 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852308 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.852717 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852850 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.852911 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.853117 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.853194 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:00:15.853366 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.849711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:35555 (local address 127.2.208.131:57597)
0620 19:00:15.849949 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.849966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.850090 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.850365 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.850368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.850384 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.850545 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.850555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.851333 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.851337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.852182 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.852190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.852840 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.852992 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.853021 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.853117 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.853211 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":53,"thread_start_us":71,"threads_started":1}
I20260620 19:00:15.853359 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.849768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:34535 (local address 127.2.208.131:35003)
0620 19:00:15.850044 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.850057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.850152 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.850430 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.850433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.850448 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:15.850597 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.850604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.851413 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.851416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.852184 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.852191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.852841 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.852844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.852998 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.853016 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.853077 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.853193 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260620 19:00:15.854048 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5d42adcfa4498999a72f727cb6cdc3" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a1255df743490099d44aa2ebe98fc7" is_pre_election: true
I20260620 19:00:15.854296 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5d42adcfa4498999a72f727cb6cdc3" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a" is_pre_election: true
I20260620 19:00:15.854319 4154 raft_consensus.cc:2454] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 0.
I20260620 19:00:15.854444 4294 raft_consensus.cc:2454] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 0.
I20260620 19:00:15.854712 4366 leader_election.cc:302] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2217ca570cf34d2ea5eb32ef66dccd71, d3a1255df743490099d44aa2ebe98fc7, fa9ed448f38143ec82eca1a161aab24a; no voters:
I20260620 19:00:15.855002 4511 raft_consensus.cc:2781] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:15.855060 4511 raft_consensus.cc:482] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:15.855104 4511 raft_consensus.cc:3037] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.856115 4511 raft_consensus.cc:504] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.856451 4511 leader_election.cc:288] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 election: Requested vote from peers d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555), fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535)
I20260620 19:00:15.856736 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5d42adcfa4498999a72f727cb6cdc3" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a1255df743490099d44aa2ebe98fc7"
I20260620 19:00:15.856774 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5d42adcfa4498999a72f727cb6cdc3" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a"
I20260620 19:00:15.856863 4154 raft_consensus.cc:3037] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.856899 4294 raft_consensus.cc:3037] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:15.857751 4154 raft_consensus.cc:2454] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 1.
I20260620 19:00:15.857789 4294 raft_consensus.cc:2454] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 1.
I20260620 19:00:15.858006 4366 leader_election.cc:302] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2217ca570cf34d2ea5eb32ef66dccd71, d3a1255df743490099d44aa2ebe98fc7; no voters:
I20260620 19:00:15.858238 4511 raft_consensus.cc:2781] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:15.858395 4511 raft_consensus.cc:686] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 LEADER]: Becoming Leader. State: Replica: 2217ca570cf34d2ea5eb32ef66dccd71, State: Running, Role: LEADER
I20260620 19:00:15.858639 4511 consensus_queue.cc:237] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.860651 3997 catalog_manager.cc:5495] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } health_report { overall_health: HEALTHY } } }
Jun 20 19:00:15 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982014, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:15.921818 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.912955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39029 (local address 127.0.0.1:36260)
0620 19:00:15.913693 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:15.913712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:15.913827 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:15.914073 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:15.914075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:15.914411 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:15.914505 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.914520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.915599 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.915602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.916137 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:15.916144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.916719 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.916721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:15.916783 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.917398 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:15.917409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:15.918715 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:15.920473 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.920480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.920484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.920700 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.920959 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:15.920961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:15.920962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:15.921034 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:15.921388 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:15.921393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:15.921480 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:00:15.921575 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:00:15.921672 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260620 19:00:15.921967 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:15.913056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.2.208.190:39029)
0620 19:00:15.913317 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:15.913325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:15.913803 (+ 478us) server_negotiation.cc:407] Connection header received
0620 19:00:15.913923 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:15.913926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:15.913970 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:15.914058 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:15.914620 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.915494 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.916260 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:15.916608 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:15.916703 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:15.918834 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:15.918838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:15.918841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:15.918861 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:15.920375 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.920791 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.920794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.920796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.920874 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:15.921112 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:15.921114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:15.921116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:15.921295 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:15.921384 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:15.921653 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:00:15.921782 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":128,"threads_started":1}
I20260620 19:00:15.926750 3997 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:36260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:00:15.927554 3997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:15.932050 4134 tablet_service.cc:1467] Processing CreateTablet for tablet c8c5d69302fc4f83a07a6b536fe1ee6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89fe8ddecef34be784480cdd40eae0a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:15.932078 4274 tablet_service.cc:1467] Processing CreateTablet for tablet c8c5d69302fc4f83a07a6b536fe1ee6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89fe8ddecef34be784480cdd40eae0a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:15.932245 4414 tablet_service.cc:1467] Processing CreateTablet for tablet c8c5d69302fc4f83a07a6b536fe1ee6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89fe8ddecef34be784480cdd40eae0a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:15.932444 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c5d69302fc4f83a07a6b536fe1ee6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.932446 4134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c5d69302fc4f83a07a6b536fe1ee6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.932576 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c5d69302fc4f83a07a6b536fe1ee6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:15.934796 4503 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: Bootstrap starting.
I20260620 19:00:15.935132 4504 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: Bootstrap starting.
I20260620 19:00:15.935655 4505 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: Bootstrap starting.
I20260620 19:00:15.935691 4503 tablet_bootstrap.cc:654] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.936019 4504 tablet_bootstrap.cc:654] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.936480 4505 tablet_bootstrap.cc:654] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:15.936983 4503 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: No bootstrap required, opened a new log
I20260620 19:00:15.937062 4503 ts_tablet_manager.cc:1397] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:15.937218 4504 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: No bootstrap required, opened a new log
I20260620 19:00:15.937297 4504 ts_tablet_manager.cc:1397] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:15.937503 4503 raft_consensus.cc:348] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.937613 4503 raft_consensus.cc:374] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.937633 4503 raft_consensus.cc:729] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9ed448f38143ec82eca1a161aab24a, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.937747 4505 tablet_bootstrap.cc:492] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: No bootstrap required, opened a new log
I20260620 19:00:15.937726 4504 raft_consensus.cc:348] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.937757 4503 consensus_queue.cc:260] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.937813 4505 ts_tablet_manager.cc:1397] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 19:00:15.937829 4504 raft_consensus.cc:374] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.937861 4504 raft_consensus.cc:729] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a1255df743490099d44aa2ebe98fc7, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.938043 4503 ts_tablet_manager.cc:1428] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:15.937980 4504 consensus_queue.cc:260] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.937984 4510 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:15.938127 4510 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:15.938268 4504 ts_tablet_manager.cc:1428] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:15.938264 4505 raft_consensus.cc:348] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.938367 4505 raft_consensus.cc:374] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:15.938424 4505 raft_consensus.cc:729] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2217ca570cf34d2ea5eb32ef66dccd71, State: Initialized, Role: FOLLOWER
I20260620 19:00:15.938529 4505 consensus_queue.cc:260] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:15.938771 4511 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:15.938900 4511 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:15.938900 4514 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:15.939033 4514 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:15.938783 4505 ts_tablet_manager.cc:1428] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:16.014736 4511 raft_consensus.cc:482] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:16.014861 4511 raft_consensus.cc:504] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:16.015296 4511 leader_election.cc:288] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555), fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535)
I20260620 19:00:16.015731 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c5d69302fc4f83a07a6b536fe1ee6c" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a1255df743490099d44aa2ebe98fc7" is_pre_election: true
I20260620 19:00:16.015745 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c5d69302fc4f83a07a6b536fe1ee6c" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a" is_pre_election: true
I20260620 19:00:16.015951 4294 raft_consensus.cc:2454] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 0.
I20260620 19:00:16.015955 4154 raft_consensus.cc:2454] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 0.
I20260620 19:00:16.016220 4367 leader_election.cc:302] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2217ca570cf34d2ea5eb32ef66dccd71, fa9ed448f38143ec82eca1a161aab24a; no voters:
I20260620 19:00:16.016407 4511 raft_consensus.cc:2781] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:16.016482 4511 raft_consensus.cc:482] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:16.016531 4511 raft_consensus.cc:3037] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:16.017525 4511 raft_consensus.cc:504] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:16.017906 4511 leader_election.cc:288] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 election: Requested vote from peers d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555), fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535)
I20260620 19:00:16.018219 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c5d69302fc4f83a07a6b536fe1ee6c" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a1255df743490099d44aa2ebe98fc7"
I20260620 19:00:16.018340 4154 raft_consensus.cc:3037] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:16.018307 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c5d69302fc4f83a07a6b536fe1ee6c" candidate_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9ed448f38143ec82eca1a161aab24a"
I20260620 19:00:16.018421 4294 raft_consensus.cc:3037] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:16.019261 4154 raft_consensus.cc:2454] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 1.
I20260620 19:00:16.019436 4294 raft_consensus.cc:2454] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2217ca570cf34d2ea5eb32ef66dccd71 in term 1.
I20260620 19:00:16.019598 4366 leader_election.cc:302] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2217ca570cf34d2ea5eb32ef66dccd71, d3a1255df743490099d44aa2ebe98fc7; no voters:
I20260620 19:00:16.019757 4511 raft_consensus.cc:2781] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:16.019870 4511 raft_consensus.cc:686] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 LEADER]: Becoming Leader. State: Replica: 2217ca570cf34d2ea5eb32ef66dccd71, State: Running, Role: LEADER
I20260620 19:00:16.020036 4511 consensus_queue.cc:237] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } }
I20260620 19:00:16.020684 4534 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: TxnStatusTablet state changed. Reason: New leader 2217ca570cf34d2ea5eb32ef66dccd71. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.020820 4534 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:16.021126 4547 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:00:16.021317 4547 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:00:16.021751 3996 catalog_manager.cc:5495] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } health_report { overall_health: HEALTHY } } }
I20260620 19:00:16.041239 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.036817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.2.208.131:44381)
0620 19:00:16.037097 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:16.037105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:16.037122 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:16.037178 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:16.037180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:16.037225 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:16.037353 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:16.037630 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.038765 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.039780 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.040008 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.040118 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.040416 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:16.040829 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:16.040952 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:16.041033 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:00:16.041083 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260620 19:00:16.041222 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.036681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44381 (local address 127.0.0.1:38758)
0620 19:00:16.036826 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:16.036871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:16.036979 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:16.037375 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:16.037378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:16.037385 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:16.037492 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:16.037497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.038880 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.038883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:16.039669 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:16.039678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.040117 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.040121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:16.040214 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.040264 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:16.040931 (+ 667us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:16.040946 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:00:16.041023 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:00:16.041103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:00:16.043989 4294 raft_consensus.cc:1261] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Refusing update from remote peer 2217ca570cf34d2ea5eb32ef66dccd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:16.044180 4154 raft_consensus.cc:1261] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Refusing update from remote peer 2217ca570cf34d2ea5eb32ef66dccd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:16.044551 4534 consensus_queue.cc:1035] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:16.044704 4511 consensus_queue.cc:1035] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:16.046365 4510 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: TxnStatusTablet state changed. Reason: New leader 2217ca570cf34d2ea5eb32ef66dccd71. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.046485 4510 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:16.046463 4514 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: TxnStatusTablet state changed. Reason: New leader 2217ca570cf34d2ea5eb32ef66dccd71. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.046594 4514 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:16.049085 4534 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.049176 4511 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.049276 4511 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:16.049389 4518 mvcc.cc:205] Tried to move back new op lower bound from 7298998337711374336 to 7298998337619308544. Current Snapshot: MvccSnapshot[applied={T|T < 7298998337711374336 or (T in {7298998337711374336})}]
I20260620 19:00:16.049621 4510 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.049718 4510 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P fa9ed448f38143ec82eca1a161aab24a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:16.049189 4534 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P 2217ca570cf34d2ea5eb32ef66dccd71: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:16.050606 4514 tablet_replica.cc:439] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 } } peers { permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 } } peers { permanent_uuid: "2217ca570cf34d2ea5eb32ef66dccd71" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44381 } } }
I20260620 19:00:16.050712 4514 tablet_replica.cc:442] T c8c5d69302fc4f83a07a6b536fe1ee6c P d3a1255df743490099d44aa2ebe98fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:16.057529 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.054392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.2.208.131:44381)
0620 19:00:16.054678 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:16.054681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:16.054690 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:00:16.054722 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:16.054724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:16.054756 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:16.054846 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:16.055091 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.055672 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.056487 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.056712 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.056823 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.057071 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:16.057139 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:16.057222 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:16.057345 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:00:16.057392 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260620 19:00:16.067688 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.064369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:52099 (local address 127.2.208.131:44381)
0620 19:00:16.064484 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:16.064490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:16.064684 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:00:16.064817 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:16.064820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:16.064859 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:16.064938 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:16.065173 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.065627 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.066558 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.066836 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.066940 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.067233 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:16.067296 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:16.067373 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:16.067498 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:00:16.067574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":29}
I20260620 19:00:16.067997 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.064303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44381 (local address 127.2.208.131:52099)
0620 19:00:16.064601 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:16.064618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:16.064728 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:16.064944 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:16.064947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:16.064954 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:16.065063 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:16.065068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.065730 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.065736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:16.066443 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:16.066456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.066937 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.066941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:16.067075 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.067119 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:16.067423 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:16.067429 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:00:16.067490 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:00:16.067883 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":37,"thread_start_us":106,"threads_started":1}
I20260620 19:00:16.074429 4154 raft_consensus.cc:1261] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Refusing update from remote peer 2217ca570cf34d2ea5eb32ef66dccd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:16.074429 4294 raft_consensus.cc:1261] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Refusing update from remote peer 2217ca570cf34d2ea5eb32ef66dccd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:16.074815 4553 consensus_queue.cc:1035] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9ed448f38143ec82eca1a161aab24a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:16.074949 4534 consensus_queue.cc:1035] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a1255df743490099d44aa2ebe98fc7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:16.112498 3995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:16.112694 3995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:16.114675 3995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadba9a16efc4261bae2af707475459a: Sending DeleteTablet for 3 replicas of tablet 7c5d42adcfa4498999a72f727cb6cdc3
I20260620 19:00:16.115206 4134 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:16 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50562
I20260620 19:00:16.115274 4274 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:16 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:54078
I20260620 19:00:16.115585 4561 tablet_replica.cc:330] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: stopping tablet replica
I20260620 19:00:16.115722 4561 raft_consensus.cc:2229] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:16.115849 4561 raft_consensus.cc:2258] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:16.116555 4561 ts_tablet_manager.cc:1905] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:16.116758 4413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5d42adcfa4498999a72f727cb6cdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:16 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:38742
I20260620 19:00:16.117166 4560 tablet_replica.cc:330] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: stopping tablet replica
I20260620 19:00:16.117296 4560 raft_consensus.cc:2229] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:16.117417 4560 raft_consensus.cc:2258] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:16.118067 4560 ts_tablet_manager.cc:1905] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:16.119771 4561 ts_tablet_manager.cc:1918] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:16.119843 4561 log.cc:1196] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/7c5d42adcfa4498999a72f727cb6cdc3
I20260620 19:00:16.120124 4561 ts_tablet_manager.cc:1939] T 7c5d42adcfa4498999a72f727cb6cdc3 P fa9ed448f38143ec82eca1a161aab24a: Deleting consensus metadata
I20260620 19:00:16.120779 3983 catalog_manager.cc:4841] TS fa9ed448f38143ec82eca1a161aab24a (127.2.208.130:34535): tablet 7c5d42adcfa4498999a72f727cb6cdc3 (table test-table [id=4d528e0c2a90491b850c791fc8c9b172]) successfully deleted
I20260620 19:00:16.121075 4563 tablet_replica.cc:330] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: stopping tablet replica
I20260620 19:00:16.121093 4560 ts_tablet_manager.cc:1918] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:16.121166 4560 log.cc:1196] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/7c5d42adcfa4498999a72f727cb6cdc3
I20260620 19:00:16.121204 4563 raft_consensus.cc:2229] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:16.121464 4560 ts_tablet_manager.cc:1939] T 7c5d42adcfa4498999a72f727cb6cdc3 P d3a1255df743490099d44aa2ebe98fc7: Deleting consensus metadata
I20260620 19:00:16.121523 4563 raft_consensus.cc:2258] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:16.122115 3982 catalog_manager.cc:4841] TS d3a1255df743490099d44aa2ebe98fc7 (127.2.208.129:35555): tablet 7c5d42adcfa4498999a72f727cb6cdc3 (table test-table [id=4d528e0c2a90491b850c791fc8c9b172]) successfully deleted
I20260620 19:00:16.122335 4563 ts_tablet_manager.cc:1905] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:16.125332 4563 ts_tablet_manager.cc:1918] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:16.125433 4563 log.cc:1196] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/7c5d42adcfa4498999a72f727cb6cdc3
Jun 20 19:00:16 dist-test-slave-tsm4 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:16.125679 4563 ts_tablet_manager.cc:1939] T 7c5d42adcfa4498999a72f727cb6cdc3 P 2217ca570cf34d2ea5eb32ef66dccd71: Deleting consensus metadata
I20260620 19:00:16.126300 3983 catalog_manager.cc:4841] TS 2217ca570cf34d2ea5eb32ef66dccd71 (127.2.208.131:44381): tablet 7c5d42adcfa4498999a72f727cb6cdc3 (table test-table [id=4d528e0c2a90491b850c791fc8c9b172]) successfully deleted
I20260620 19:00:16.129959 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.116952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.2.208.129:35555)
0620 19:00:16.117064 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:16.117069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:16.117659 (+ 590us) server_negotiation.cc:407] Connection header received
0620 19:00:16.117849 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:16.117852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:16.117895 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:16.117944 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:16.118979 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.119862 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.121361 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.121674 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.121751 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.126387 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:16.126400 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:16.126417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:16.126445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:16.128207 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:16.128714 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:16.128718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:16.128720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:16.128793 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:16.129069 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:16.129071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:16.129072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:16.129306 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:16.129478 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:16.129698 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 19:00:16.129790 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 19:00:16.130431 4194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50578
I20260620 19:00:16.140558 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:16.132113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.2.208.190:39029)
0620 19:00:16.132236 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:16.132240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:16.132325 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:00:16.132398 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:16.132401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:16.132446 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:16.132527 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:16.133083 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.133903 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.135002 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:16.135354 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:16.135473 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:16.136939 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:16.136943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:16.136945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:16.136969 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:16.138726 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:16.139230 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:16.139234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:16.139235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:16.139319 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:16.139639 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:16.139642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:16.139644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:16.139845 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:16.139983 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:16.140227 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:00:16.140396 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:00:16.140884 3995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36266
I20260620 19:00:16.142273 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4069
I20260620 19:00:16.151463 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4209
I20260620 19:00:16.158644 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4349
I20260620 19:00:16.164860 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 3965
2026-06-20T19:00:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3068 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:16 dist-test-slave-tsm4 krb5kdc[4569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:16 dist-test-slave-tsm4 krb5kdc[4569](info): set up 2 sockets
Jun 20 19:00:16 dist-test-slave-tsm4 krb5kdc[4569](info): commencing operation
krb5kdc: starting...
W20260620 19:00:17.199182 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:17.318662 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43167
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:17.425242 4582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:17.425526 4582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:17.425606 4582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:17.429545 4582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:17.429629 4582 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:00:17.429661 4582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:17.429716 4582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:17.429749 4582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:17.429780 4582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:17.434932 4582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43167
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:17.436259 4582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:17.437268 4582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:17.445472 4587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:17.445515 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:17.445573 4582 server_base.cc:1034] running on GCE node
W20260620 19:00:17.445705 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:17.446255 4582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:17.447275 4582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:17.448469 4582 hybrid_clock.cc:648] HybridClock initialized: now 1781982017448455 us; error 26 us; skew 500 ppm
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:17.451131 4582 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:17.451987 4582 webserver.cc:460] Webserver started at http://127.2.208.190:37599/ using document root <none> and password file <none>
I20260620 19:00:17.452548 4582 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:17.452643 4582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:17.452863 4582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:17.454953 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "779cc85e0ee64783904e6a495a0a1a11"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "a7b6e4dde6c869f1f3a02bf6dc9b01c9"
server_key_iv: "42f5ac5149901cfce67b19616969836f"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.455567 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "779cc85e0ee64783904e6a495a0a1a11"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "a7b6e4dde6c869f1f3a02bf6dc9b01c9"
server_key_iv: "42f5ac5149901cfce67b19616969836f"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.459017 4582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:17.461938 4597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:17.463287 4582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:00:17.463433 4582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "779cc85e0ee64783904e6a495a0a1a11"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "a7b6e4dde6c869f1f3a02bf6dc9b01c9"
server_key_iv: "42f5ac5149901cfce67b19616969836f"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.463541 4582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:17.486519 4582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:17.489419 4582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:17.489622 4582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:17.497148 4582 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:43167
I20260620 19:00:17.497155 4659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:43167 every 8 connection(s)
I20260620 19:00:17.498013 4582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:17.500658 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:17.504663 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4582
I20260620 19:00:17.504783 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:17.505048 2882 external_mini_cluster.cc:1428] Setting key 8d9ccef7cce243dbd98a01dcf6b12be3
I20260620 19:00:17.506285 4660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Bootstrap starting.
I20260620 19:00:17.508643 4660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:17.509409 4660 log.cc:824] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:17.511336 4660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: No bootstrap required, opened a new log
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:17.514974 4660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } }
I20260620 19:00:17.515276 4660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:17.515333 4660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779cc85e0ee64783904e6a495a0a1a11, State: Initialized, Role: FOLLOWER
I20260620 19:00:17.515830 4660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } }
I20260620 19:00:17.515987 4660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:17.516052 4660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:17.516152 4660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:17.517234 4660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } }
I20260620 19:00:17.517580 4660 leader_election.cc:302] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779cc85e0ee64783904e6a495a0a1a11; no voters:
I20260620 19:00:17.517926 4660 leader_election.cc:288] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:17.518074 4665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:17.518337 4665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [term 1 LEADER]: Becoming Leader. State: Replica: 779cc85e0ee64783904e6a495a0a1a11, State: Running, Role: LEADER
I20260620 19:00:17.518633 4665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } }
I20260620 19:00:17.519004 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.506329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.2.208.190:43167)
0620 19:00:17.506740 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:17.506755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:17.506787 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:00:17.507425 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:17.507440 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:17.507750 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:17.508007 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:17.508888 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.509915 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.510815 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.511523 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.511687 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.514030 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:17.514037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:17.514062 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:17.514093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:17.516062 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.516505 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.516510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.516516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.516642 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.516899 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.516905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.516907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.517406 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:17.517576 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:17.517854 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:00:17.518163 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260620 19:00:17.519053 4660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:17.520165 4666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "779cc85e0ee64783904e6a495a0a1a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } } }
I20260620 19:00:17.520283 4666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:17.520530 4666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779cc85e0ee64783904e6a495a0a1a11. Latest consensus state: current_term: 1 leader_uuid: "779cc85e0ee64783904e6a495a0a1a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779cc85e0ee64783904e6a495a0a1a11" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43167 } } }
I20260620 19:00:17.520614 4666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11 [sys.catalog]: This master's current role is: LEADER
W20260620 19:00:17.523718 4680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:17.523779 4680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:17.523888 4674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:17.524591 4674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:17.529096 4674 catalog_manager.cc:1347] Generated new cluster ID: fe90e75dbad64b52b9223d2cae36ecc5
I20260620 19:00:17.529155 4674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:17.542068 4674 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:17.542925 4674 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:17.548792 4674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Generated new TSK 0
I20260620 19:00:17.549480 4674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:17.582304 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43167
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:17.689040 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:17.689275 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:17.689318 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:17.693272 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:17.693322 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:17.693389 4686 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:00:17.693424 4686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:17.693478 4686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:17.699172 4686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:17.700119 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:17.701073 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:17.710359 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:17.710366 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:17.710361 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:17.710659 4686 server_base.cc:1034] running on GCE node
I20260620 19:00:17.711086 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:17.711752 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:17.712950 4686 hybrid_clock.cc:648] HybridClock initialized: now 1781982017712918 us; error 41 us; skew 500 ppm
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:17.715624 4686 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:17.716388 4686 webserver.cc:460] Webserver started at http://127.2.208.129:39103/ using document root <none> and password file <none>
I20260620 19:00:17.716984 4686 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:17.717074 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:17.717273 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:17.719596 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "a761a21430de4522ab92832bf038148e"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "fb77be10ca92566ada3d7f070567227c"
server_key_iv: "f7732d1f61e5da7dd83bd339609c62b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.720301 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "a761a21430de4522ab92832bf038148e"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "fb77be10ca92566ada3d7f070567227c"
server_key_iv: "f7732d1f61e5da7dd83bd339609c62b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.723904 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:17.726286 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:17.727563 4686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:17.727691 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "a761a21430de4522ab92832bf038148e"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "fb77be10ca92566ada3d7f070567227c"
server_key_iv: "f7732d1f61e5da7dd83bd339609c62b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.727797 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:17.745829 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:17.748584 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:17.748777 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:17.750099 4686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:17.750174 4686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:17.750237 4686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:17.750294 4686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:17.760643 4686 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:33127
I20260620 19:00:17.760694 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:33127 every 8 connection(s)
I20260620 19:00:17.761660 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:17.764246 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.750986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37791 (local address 127.2.208.190:43167)
0620 19:00:17.751136 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:17.751140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:17.752071 (+ 931us) server_negotiation.cc:407] Connection header received
0620 19:00:17.753347 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:17.753351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:17.753413 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:17.753545 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:17.755028 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.755883 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.757739 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.757974 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.758062 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.760588 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:17.760592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:17.760595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:17.760631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:17.762183 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.762745 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.762748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.762750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.762833 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.763192 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.763195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.763197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.763416 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:17.763504 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:17.763967 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 19:00:17.764089 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 19:00:17.764885 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.751410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.129:37791)
0620 19:00:17.751907 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:17.751958 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:17.753104 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:17.753728 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:17.753741 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:17.754187 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:17.754858 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.754875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.756075 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.756080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.757622 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.757635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.758157 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.758161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.758349 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.758938 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:17.758955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:17.760431 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:17.762309 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.762315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.762328 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.762655 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.762928 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.762932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.762934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.763084 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.763501 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:17.763508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:17.763728 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 19:00:17.763915 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 19:00:17.764192 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260620 19:00:17.766189 4817 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43167
I20260620 19:00:17.766345 4817 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:17.766788 4817 heartbeater.cc:511] Master 127.2.208.190:43167 requested a full tablet report, sending...
I20260620 19:00:17.767771 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4686
I20260620 19:00:17.767882 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:17.768190 2882 external_mini_cluster.cc:1428] Setting key d15d943ae0b87c40f017552d2f4d0856
I20260620 19:00:17.768929 4613 ts_manager.cc:194] Registered new tserver with Master: a761a21430de4522ab92832bf038148e (127.2.208.129:33127)
I20260620 19:00:17.769922 4613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:37791
I20260620 19:00:17.778141 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.770397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:54875 (local address 127.2.208.190:43167)
0620 19:00:17.770509 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:17.770512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:17.771281 (+ 769us) server_negotiation.cc:407] Connection header received
0620 19:00:17.771421 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:17.771424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:17.771466 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:17.771549 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:17.771939 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.772408 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.773630 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.773833 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.773913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.775036 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:17.775039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:17.775041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:17.775060 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:17.776578 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.777069 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.777072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.777073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.777138 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.777433 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.777435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.777437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.777589 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:17.777658 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:17.777956 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 19:00:17.778040 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:17.778252 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.768294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.129:54875)
0620 19:00:17.771163 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:17.771180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:17.771317 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:17.771570 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:17.771573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:17.771746 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:17.771833 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.771837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.772526 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.772529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.773489 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.773502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.773949 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.773953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.774047 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.774432 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:17.774451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:17.774889 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:17.776691 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.776697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.776701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.776965 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.777235 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.777241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.777243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.777341 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.777687 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:17.777690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:17.777789 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:00:17.777909 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:00:17.778037 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2787,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:17.798134 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43167
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:17.902987 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:17.903204 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:17.903246 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:17.907315 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:17.907423 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:17.907465 4826 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:00:17.907493 4826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:17.907568 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:17.912606 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:17.913501 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:17.914413 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:17.923552 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:17.923625 4834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:17.923576 4826 server_base.cc:1034] running on GCE node
W20260620 19:00:17.923717 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:17.924172 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:17.924674 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:17.925833 4826 hybrid_clock.cc:648] HybridClock initialized: now 1781982017925815 us; error 28 us; skew 500 ppm
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:17.928581 4826 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:17.929360 4826 webserver.cc:460] Webserver started at http://127.2.208.130:37339/ using document root <none> and password file <none>
I20260620 19:00:17.929841 4826 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:17.929895 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:17.930109 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:17.932097 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "7be750056bc44b3a81394a9663938701"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "48439d36616ad97afc2645c431a76044"
server_key_iv: "1590b9a9ea25323842e0bfa706f28605"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.932572 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "7be750056bc44b3a81394a9663938701"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "48439d36616ad97afc2645c431a76044"
server_key_iv: "1590b9a9ea25323842e0bfa706f28605"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.935914 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:17.938063 4841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:17.939205 4826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:17.939320 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "7be750056bc44b3a81394a9663938701"
format_stamp: "Formatted at 2026-06-20 19:00:17 on dist-test-slave-tsm4"
server_key: "48439d36616ad97afc2645c431a76044"
server_key_iv: "1590b9a9ea25323842e0bfa706f28605"
server_key_version: "encryptionkey@0"
I20260620 19:00:17.939468 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:17.958420 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:17.961611 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:17.961814 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:17.963158 4826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:17.963238 4826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:17.963303 4826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:17.963357 4826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:17 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:17.973897 4826 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:37813
I20260620 19:00:17.974190 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:37813 every 8 connection(s)
I20260620 19:00:17.974802 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:17.976706 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:37719 (local address 127.2.208.190:43167)
0620 19:00:17.964152 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:17.964157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:17.964986 (+ 829us) server_negotiation.cc:407] Connection header received
0620 19:00:17.965972 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:17.965976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:17.966037 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:17.966098 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:17.967714 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.968636 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.970064 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.970398 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.970523 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.973068 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:17.973072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:17.973075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:17.973100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:17.974605 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.975158 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.975163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.975165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.975283 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.975838 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.975840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.975841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.976015 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:17.976097 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:17.976502 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 19:00:17.976595 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:00:17.977398 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.964402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.130:37719)
0620 19:00:17.964825 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:17.964872 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:17.965734 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:17.966509 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:17.966516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:17.966978 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:17.967566 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.967586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.968759 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.968762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.969936 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.969945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.970569 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.970573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.970737 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.971266 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:17.971279 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:17.972918 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:17.974717 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.974722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.974734 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.975058 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.975413 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.975416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.975418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.975589 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.976110 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:17.976115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:17.976301 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 19:00:17.976433 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 19:00:17.976643 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":126,"threads_started":1}
I20260620 19:00:17.978541 4957 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43167
I20260620 19:00:17.978701 4957 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:17.979151 4957 heartbeater.cc:511] Master 127.2.208.190:43167 requested a full tablet report, sending...
I20260620 19:00:17.980152 4614 ts_manager.cc:194] Registered new tserver with Master: 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813)
I20260620 19:00:17.980803 4614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:37719
I20260620 19:00:17.983997 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4826
I20260620 19:00:17.984093 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:17.984387 2882 external_mini_cluster.cc:1428] Setting key 6269b71c4b40f350d60c6fee1b8d4a6e
I20260620 19:00:17.987335 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.980537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59537 (local address 127.2.208.190:43167)
0620 19:00:17.980689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:17.980694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:17.980892 (+ 198us) server_negotiation.cc:407] Connection header received
0620 19:00:17.981022 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:17.981025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:17.981076 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:17.981158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:17.981576 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.982055 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.982861 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.983080 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.983191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.984280 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:17.984283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:17.984285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:17.984307 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:17.985699 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.986105 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.986109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.986111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.986186 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:17.986468 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:17.986471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:17.986472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:17.986651 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:17.986757 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:17.987039 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:00:17.987154 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 19:00:17.987839 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:17.980473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.130:59537)
0620 19:00:17.980810 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:17.980820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:17.980925 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:17.981172 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:17.981174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:17.981352 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:17.981441 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.981465 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.982183 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.982186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.982733 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:17.982741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:17.983194 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:17.983197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:17.983287 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:17.983815 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:17.983826 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:17.984170 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:17.985823 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.985827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.985829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.986008 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.986288 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:17.986291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:17.986293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:17.986373 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:17.986749 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:17.986752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:17.986841 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:00:17.987480 (+ 639us) client_negotiation.cc:240] Negotiation successful
0620 19:00:17.987637 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:18.009744 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43167
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:18.116645 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:18.116861 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:18.116904 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:18.121104 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:18.121155 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:18.121222 4966 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:00:18.121255 4966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:18.121332 4966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:18.126619 4966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:18.127599 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:18.128535 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:18.137816 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:18.137811 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:18.137883 4966 server_base.cc:1034] running on GCE node
W20260620 19:00:18.137810 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:18.138428 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:18.138953 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:18.140105 4966 hybrid_clock.cc:648] HybridClock initialized: now 1781982018140078 us; error 41 us; skew 500 ppm
Jun 20 19:00:18 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982018, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:18.142766 4966 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:18.143663 4966 webserver.cc:460] Webserver started at http://127.2.208.131:35209/ using document root <none> and password file <none>
I20260620 19:00:18.144178 4966 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:18.144235 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:18.144464 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:18.146286 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
format_stamp: "Formatted at 2026-06-20 19:00:18 on dist-test-slave-tsm4"
server_key: "874118d7116d5f7e3d4fa2c74e98f75e"
server_key_iv: "5285f05ca5ec53cbcbfaf7c5814a95fc"
server_key_version: "encryptionkey@0"
I20260620 19:00:18.146806 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
format_stamp: "Formatted at 2026-06-20 19:00:18 on dist-test-slave-tsm4"
server_key: "874118d7116d5f7e3d4fa2c74e98f75e"
server_key_iv: "5285f05ca5ec53cbcbfaf7c5814a95fc"
server_key_version: "encryptionkey@0"
I20260620 19:00:18.150336 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:18.152509 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:18.153695 4966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:18.153827 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
format_stamp: "Formatted at 2026-06-20 19:00:18 on dist-test-slave-tsm4"
server_key: "874118d7116d5f7e3d4fa2c74e98f75e"
server_key_iv: "5285f05ca5ec53cbcbfaf7c5814a95fc"
server_key_version: "encryptionkey@0"
I20260620 19:00:18.153934 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:18.174203 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:18.177153 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:18.177353 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:18.178905 4966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:18.178973 4966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:18.179041 4966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:18.179075 4966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:18 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982018, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:18.190156 4966 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:44815
I20260620 19:00:18.190246 5096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:44815 every 8 connection(s)
I20260620 19:00:18.191066 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:18.192698 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.179650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:52295 (local address 127.2.208.190:43167)
0620 19:00:18.179763 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.179767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.180797 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 19:00:18.181879 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.181884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.181949 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.181997 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:18.183385 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.184296 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.185598 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.185903 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.186031 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.188867 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:18.188871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:18.188874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:18.188907 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:18.190558 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.191077 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.191081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.191082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.191171 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.191595 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.191598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.191600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.191837 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:18.191939 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.192449 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.192580 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:00:18.193506 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.180189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.131:52295)
0620 19:00:18.180654 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.180693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.181636 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.182390 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.182396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.182700 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:18.183189 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.183207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.184447 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.184451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.185459 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.185468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.186081 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.186085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.186238 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.186842 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:18.186871 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:18.188693 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:18.190676 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.190683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.190700 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.190962 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.191275 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.191279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.191281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.191479 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.191962 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:18.191967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:18.192218 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.192379 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.192635 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260620 19:00:18.194804 5097 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43167
I20260620 19:00:18.194974 5097 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:18.195580 5097 heartbeater.cc:511] Master 127.2.208.190:43167 requested a full tablet report, sending...
I20260620 19:00:18.195665 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 4966
I20260620 19:00:18.195777 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:18.196065 2882 external_mini_cluster.cc:1428] Setting key ad6b32fd3b477554176588ed64b2dd74
I20260620 19:00:18.196606 4614 ts_manager.cc:194] Registered new tserver with Master: b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:18.197366 4614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:52295
I20260620 19:00:18.198146 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:00:18.205363 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.197104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.2.208.131:42661)
0620 19:00:18.197502 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.197516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.197630 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.198023 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.198025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.198259 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:18.198387 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.198394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.199396 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.199403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.200317 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.200325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.200941 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.200943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.201007 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.201502 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:18.201513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:18.201894 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:18.203905 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.203908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.203910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.204199 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.204452 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.204457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.204462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.204559 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.204890 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:18.204896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:18.204998 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.205147 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.205240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
I20260620 19:00:18.205531 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.197120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:42661 (local address 127.2.208.190:43167)
0620 19:00:18.197258 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.197261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.197574 (+ 313us) server_negotiation.cc:407] Connection header received
0620 19:00:18.197730 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.197733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.197775 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.197821 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:18.198506 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.199223 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.200427 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.200832 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.200946 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.202042 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:18.202049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:18.202055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:18.202085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:18.203798 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.204295 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.204297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.204299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.204365 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.204647 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.204649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.204650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.204789 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:18.204869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.205227 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.205349 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 19:00:18 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:00:18 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:18.223954 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.216401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.2.208.190:43167)
0620 19:00:18.216565 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.216568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.216578 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:00:18.216607 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.216609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.216640 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.216743 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:18.217381 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.217862 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.218679 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.218886 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.218997 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.221261 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:18.221265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:18.221268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:18.221291 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:18.222619 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.222976 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.222979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.222980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.223050 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.223267 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.223269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.223270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.223477 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:18.223579 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.223767 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.223849 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 19:00:18.226684 4613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:18.229636 4613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:18.243609 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.237762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44815 (local address 127.0.0.1:47822)
0620 19:00:18.238126 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.238175 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.238317 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.239239 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.239246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.239268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.239430 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.239437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.240599 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.240605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.241930 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.241940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.243059 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.243062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.243176 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.243211 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.243323 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.243461 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":29,"thread_start_us":79,"threads_started":1}
I20260620 19:00:18.243618 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.237877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.2.208.131:44815)
0620 19:00:18.238458 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.238471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.238498 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:00:18.238578 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.238586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.238898 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.239064 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.239549 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.240485 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.242053 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.242904 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.243062 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.243216 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.243396 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.243479 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":125,"threads_started":1}
I20260620 19:00:18.244266 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.238805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.0.0.1:45174)
0620 19:00:18.239092 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.239106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.239191 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.239855 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.239858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.239875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.239958 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.239966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.241304 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.241310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.242436 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.242444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.243562 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.243566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.243654 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.244001 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.244067 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.244151 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":84,"thread_start_us":92,"threads_started":1}
I20260620 19:00:18.244612 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.238495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37813 (local address 127.0.0.1:39538)
0620 19:00:18.239041 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.239055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.239141 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.239997 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.240000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.240015 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.240107 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.240112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.241383 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.241386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.242544 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.242555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.243869 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.243873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.243968 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.243990 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.244051 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.244502 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":30,"thread_start_us":83,"threads_started":1}
I20260620 19:00:18.244896 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.238980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.2.208.129:33127)
0620 19:00:18.239449 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.239465 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.239489 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:00:18.239562 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.239567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.239718 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.239871 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.240049 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.241172 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.242550 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.243429 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.244158 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.244287 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.244511 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.244602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":111,"threads_started":1}
I20260620 19:00:18.244942 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.238648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.2.208.130:37813)
0620 19:00:18.239437 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.239447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.239467 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:00:18.239551 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.239556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.239736 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.239875 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.240227 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.241273 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.242912 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.243764 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.243862 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.244682 (+ 820us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.244734 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.244808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":108,"threads_started":1}
I20260620 19:00:18.246551 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 54ca0881554742eeabb3024d1fe070e7 (DEFAULT_TABLE table=test-table [id=8e6e97f323e743178b19d4f2cb47b425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.246989 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 54ca0881554742eeabb3024d1fe070e7 (DEFAULT_TABLE table=test-table [id=8e6e97f323e743178b19d4f2cb47b425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.247653 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ca0881554742eeabb3024d1fe070e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.247948 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ca0881554742eeabb3024d1fe070e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.247745 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 54ca0881554742eeabb3024d1fe070e7 (DEFAULT_TABLE table=test-table [id=8e6e97f323e743178b19d4f2cb47b425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.248684 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ca0881554742eeabb3024d1fe070e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.254798 5120 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Bootstrap starting.
I20260620 19:00:18.255525 5122 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Bootstrap starting.
I20260620 19:00:18.256228 5121 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Bootstrap starting.
I20260620 19:00:18.257031 5120 tablet_bootstrap.cc:654] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.257998 5120 log.cc:824] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:18.258112 5121 tablet_bootstrap.cc:654] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.258891 5121 log.cc:824] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:18.259027 5122 tablet_bootstrap.cc:654] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.259759 5122 log.cc:824] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:18.260140 5120 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: No bootstrap required, opened a new log
I20260620 19:00:18.260327 5120 ts_tablet_manager.cc:1397] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:00:18.260737 5121 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: No bootstrap required, opened a new log
I20260620 19:00:18.260905 5121 ts_tablet_manager.cc:1397] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:18.261348 5122 tablet_bootstrap.cc:492] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: No bootstrap required, opened a new log
I20260620 19:00:18.261521 5122 ts_tablet_manager.cc:1397] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 19:00:18.264228 5120 raft_consensus.cc:348] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.264484 5120 raft_consensus.cc:374] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.264546 5120 raft_consensus.cc:729] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a761a21430de4522ab92832bf038148e, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.264511 5121 raft_consensus.cc:348] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.264513 5122 raft_consensus.cc:348] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.264761 5122 raft_consensus.cc:374] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.264775 5121 raft_consensus.cc:374] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.264822 5121 raft_consensus.cc:729] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.264819 5122 raft_consensus.cc:729] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be750056bc44b3a81394a9663938701, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.264981 5120 consensus_queue.cc:260] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.265170 5121 consensus_queue.cc:260] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.265264 5122 consensus_queue.cc:260] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.265776 4817 heartbeater.cc:503] Master 127.2.208.190:43167 was elected leader, sending a full tablet report...
I20260620 19:00:18.265954 5121 ts_tablet_manager.cc:1428] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:00:18.265988 5120 ts_tablet_manager.cc:1428] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:00:18.266158 5122 ts_tablet_manager.cc:1428] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:00:18.266264 5097 heartbeater.cc:503] Master 127.2.208.190:43167 was elected leader, sending a full tablet report...
I20260620 19:00:18.266532 4957 heartbeater.cc:503] Master 127.2.208.190:43167 was elected leader, sending a full tablet report...
I20260620 19:00:18.381772 5127 raft_consensus.cc:482] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:18.382031 5127 raft_consensus.cc:504] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.383208 5127 leader_election.cc:288] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), a761a21430de4522ab92832bf038148e (127.2.208.129:33127)
I20260620 19:00:18.387444 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.383574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37813 (local address 127.2.208.131:41605)
0620 19:00:18.383723 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.383740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.383843 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.384214 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.384218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.384238 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.384385 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.384391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.385299 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.385302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.386149 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.386158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.386996 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.386998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.387115 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.387142 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.387220 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.387284 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 19:00:18.387472 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.383690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:41605 (local address 127.2.208.130:37813)
0620 19:00:18.383840 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.383848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.383868 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:00:18.383931 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.383937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.383990 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.384097 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.384563 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.385187 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.386290 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.386889 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.387005 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.387061 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.387248 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.387309 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:18.388175 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ca0881554742eeabb3024d1fe070e7" candidate_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701" is_pre_election: true
I20260620 19:00:18.388253 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.383869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.2.208.131:35345)
0620 19:00:18.384231 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.384242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.384322 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.384565 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.384568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.384584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.384713 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.384719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.385489 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.385492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.387017 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.387026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.387941 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.387945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.388036 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.388053 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.388115 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.388160 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
I20260620 19:00:18.388347 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.384014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:35345 (local address 127.2.208.129:33127)
0620 19:00:18.384139 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.384143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.384329 (+ 186us) server_negotiation.cc:407] Connection header received
0620 19:00:18.384389 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.384392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.384448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.384532 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.384836 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.385372 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.387143 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.387774 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.387914 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.387958 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.388158 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.388237 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:00:18.388500 4898 raft_consensus.cc:2454] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81e06e40f7d40b7af4cc98bd0a176d8 in term 0.
I20260620 19:00:18.388890 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ca0881554742eeabb3024d1fe070e7" candidate_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a761a21430de4522ab92832bf038148e" is_pre_election: true
I20260620 19:00:18.388985 4982 leader_election.cc:302] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, b81e06e40f7d40b7af4cc98bd0a176d8; no voters:
I20260620 19:00:18.389186 4771 raft_consensus.cc:2454] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81e06e40f7d40b7af4cc98bd0a176d8 in term 0.
I20260620 19:00:18.389317 5127 raft_consensus.cc:2781] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:18.389456 5127 raft_consensus.cc:482] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:18.389528 5127 raft_consensus.cc:3037] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.390520 5127 raft_consensus.cc:504] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.390839 5127 leader_election.cc:288] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [CANDIDATE]: Term 1 election: Requested vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), a761a21430de4522ab92832bf038148e (127.2.208.129:33127)
I20260620 19:00:18.391168 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ca0881554742eeabb3024d1fe070e7" candidate_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701"
I20260620 19:00:18.391275 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ca0881554742eeabb3024d1fe070e7" candidate_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a761a21430de4522ab92832bf038148e"
I20260620 19:00:18.391335 4898 raft_consensus.cc:3037] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.391436 4771 raft_consensus.cc:3037] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.392365 4898 raft_consensus.cc:2454] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81e06e40f7d40b7af4cc98bd0a176d8 in term 1.
I20260620 19:00:18.392551 4771 raft_consensus.cc:2454] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81e06e40f7d40b7af4cc98bd0a176d8 in term 1.
I20260620 19:00:18.392694 4982 leader_election.cc:302] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, b81e06e40f7d40b7af4cc98bd0a176d8; no voters:
I20260620 19:00:18.392894 5127 raft_consensus.cc:2781] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:18.393131 5127 raft_consensus.cc:686] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 LEADER]: Becoming Leader. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Running, Role: LEADER
I20260620 19:00:18.393467 5127 consensus_queue.cc:237] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } }
I20260620 19:00:18.396615 4612 catalog_manager.cc:5495] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } health_report { overall_health: UNKNOWN } } }
W20260620 19:00:18.442167 5098 tablet.cc:2369] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:00:18 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982017, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:18.452363 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.444963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43167 (local address 127.0.0.1:37388)
0620 19:00:18.445242 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.445255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.445347 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.445551 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.445553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.445918 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:18.446016 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.446022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.446723 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.446725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.447267 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.447274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.447714 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.447717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.447778 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.448287 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:18.448297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:18.449624 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:18.451016 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.451023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.451028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.451240 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.451546 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:18.451548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:18.451550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:18.451624 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:18.451943 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:18.451948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:18.452039 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.452126 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.452225 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
I20260620 19:00:18.452378 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.445010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.2.208.190:43167)
0620 19:00:18.445141 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.445146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.445324 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:00:18.445409 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.445412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.445449 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.445514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:18.446107 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.446632 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.447404 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.447618 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.447697 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.449743 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:18.449746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:18.449748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:18.449767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:18.450921 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.451324 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.451326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.451328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.451454 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:18.451703 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:18.451705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:18.451707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:18.451858 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:18.451930 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.452184 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.452268 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:18.455901 4612 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:37388:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:00:18.456707 4612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:18.461313 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9923fd19a4411b8cf90910c94d5c37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=459ad5ce13a8461fb33322f9fbc1a59b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:18.461287 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9923fd19a4411b8cf90910c94d5c37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=459ad5ce13a8461fb33322f9fbc1a59b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:18.461290 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9923fd19a4411b8cf90910c94d5c37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=459ad5ce13a8461fb33322f9fbc1a59b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:18.461971 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9923fd19a4411b8cf90910c94d5c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.461971 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9923fd19a4411b8cf90910c94d5c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.461971 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9923fd19a4411b8cf90910c94d5c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.464815 5121 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: Bootstrap starting.
I20260620 19:00:18.465128 5120 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: Bootstrap starting.
I20260620 19:00:18.465852 5121 tablet_bootstrap.cc:654] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.466449 5120 tablet_bootstrap.cc:654] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.466678 5122 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: Bootstrap starting.
I20260620 19:00:18.467344 5121 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: No bootstrap required, opened a new log
I20260620 19:00:18.467437 5121 ts_tablet_manager.cc:1397] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:00:18.467698 5122 tablet_bootstrap.cc:654] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.467932 5121 raft_consensus.cc:348] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.468055 5121 raft_consensus.cc:374] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.468089 5121 raft_consensus.cc:729] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.468217 5121 consensus_queue.cc:260] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.468360 5120 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: No bootstrap required, opened a new log
I20260620 19:00:18.468437 5120 ts_tablet_manager.cc:1397] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:18.468484 5127 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.468607 5127 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.468827 5122 tablet_bootstrap.cc:492] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: No bootstrap required, opened a new log
I20260620 19:00:18.468906 5122 ts_tablet_manager.cc:1397] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:18.468854 5120 raft_consensus.cc:348] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.469009 5120 raft_consensus.cc:374] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.469053 5120 raft_consensus.cc:729] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a761a21430de4522ab92832bf038148e, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.469187 5120 consensus_queue.cc:260] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.469310 5121 ts_tablet_manager.cc:1428] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:00:18.469319 5122 raft_consensus.cc:348] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.469449 5122 raft_consensus.cc:374] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.469491 5122 raft_consensus.cc:729] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be750056bc44b3a81394a9663938701, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.469616 5122 consensus_queue.cc:260] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.469892 5122 ts_tablet_manager.cc:1428] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:18.469852 5128 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.469964 5128 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.470224 5120 ts_tablet_manager.cc:1428] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:18.470192 5126 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.470319 5126 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 19:00:18.475937 4958 tablet.cc:2369] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:18.513041 4818 tablet.cc:2369] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:18.661898 5126 raft_consensus.cc:482] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:18.662068 5126 raft_consensus.cc:504] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.663244 5126 leader_election.cc:288] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:18.667438 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.663584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55777 (local address 127.2.208.130:37813)
0620 19:00:18.663715 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.663718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.663857 (+ 139us) server_negotiation.cc:407] Connection header received
0620 19:00:18.664031 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.664035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.664094 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.664147 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.664473 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.665041 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.666319 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.666849 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.666980 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.667032 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.667213 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.667285 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:18.667503 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.663521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37813 (local address 127.2.208.129:55777)
0620 19:00:18.663747 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.663763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.663882 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.664268 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.664271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.664297 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.664390 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.664394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.665157 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.665163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.666202 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.666212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.667007 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.667010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.667078 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.667097 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.667156 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.667262 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"spinlock_wait_cycles":5504,"thread_start_us":88,"threads_started":1}
I20260620 19:00:18.667944 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9923fd19a4411b8cf90910c94d5c37" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701" is_pre_election: true
I20260620 19:00:18.668104 4898 raft_consensus.cc:2454] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:18.668164 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.663880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44815 (local address 127.2.208.129:52625)
0620 19:00:18.664156 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.664171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.664315 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.664579 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.664581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.664595 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:18.665065 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.665070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.665803 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.665807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.666830 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.666839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.667747 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.667751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.667857 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.667878 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.667955 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.668022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"spinlock_wait_cycles":9856,"thread_start_us":66,"threads_started":1}
I20260620 19:00:18.668193 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.664012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:52625 (local address 127.2.208.131:44815)
0620 19:00:18.664134 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.664138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.664251 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:00:18.664425 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.664428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.664478 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.664552 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:18.665224 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.665690 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.666945 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.667600 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.667729 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.667773 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.667985 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.668047 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:00:18.668547 4702 leader_election.cc:302] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, a761a21430de4522ab92832bf038148e; no voters:
I20260620 19:00:18.668674 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9923fd19a4411b8cf90910c94d5c37" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" is_pre_election: true
I20260620 19:00:18.668864 5126 raft_consensus.cc:2781] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:18.668900 5051 raft_consensus.cc:2454] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:18.668956 5126 raft_consensus.cc:482] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:18.669014 5126 raft_consensus.cc:3037] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.670037 5126 raft_consensus.cc:504] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.670367 5126 leader_election.cc:288] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Requested vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:18.670663 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9923fd19a4411b8cf90910c94d5c37" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701"
I20260620 19:00:18.670769 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9923fd19a4411b8cf90910c94d5c37" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
I20260620 19:00:18.670800 4898 raft_consensus.cc:3037] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.670872 5051 raft_consensus.cc:3037] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:18.671958 4898 raft_consensus.cc:2454] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:18.672017 5051 raft_consensus.cc:2454] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:18.672237 4702 leader_election.cc:302] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a761a21430de4522ab92832bf038148e, b81e06e40f7d40b7af4cc98bd0a176d8; no voters:
I20260620 19:00:18.672411 5126 raft_consensus.cc:2781] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:18.672659 5126 raft_consensus.cc:686] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [term 1 LEADER]: Becoming Leader. State: Replica: a761a21430de4522ab92832bf038148e, State: Running, Role: LEADER
I20260620 19:00:18.672998 5126 consensus_queue.cc:237] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.674968 5143 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: TxnStatusTablet state changed. Reason: New leader a761a21430de4522ab92832bf038148e. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.675088 5143 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:18.675417 5145 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:00:18.675653 5145 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:00:18.676180 4614 catalog_manager.cc:5495] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e reported cstate change: term changed from 0 to 1, leader changed from <none> to a761a21430de4522ab92832bf038148e (127.2.208.129). New cstate: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:18.705962 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.701960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.0.0.1:45180)
0620 19:00:18.702137 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.702155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.702247 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.702577 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.702579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.702584 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:18.702669 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.702673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.703694 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.703697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.704275 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.704282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.704832 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.704834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.704897 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.704936 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.705684 (+ 748us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.705699 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.705766 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.705860 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:00:18.705965 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.702086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.2.208.129:33127)
0620 19:00:18.702204 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.702209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.702224 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:18.702389 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.702393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.702456 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.702558 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:18.702777 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.703589 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.704389 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.704726 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.704826 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.705072 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.705564 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.705674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.705775 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.705828 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:00:18.710917 4898 raft_consensus.cc:1261] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:18.710922 5051 raft_consensus.cc:1261] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:18.711628 5143 consensus_queue.cc:1035] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:18.711800 5126 consensus_queue.cc:1035] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:18.714699 5127 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: TxnStatusTablet state changed. Reason: New leader a761a21430de4522ab92832bf038148e. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.714808 5127 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.715874 5128 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: TxnStatusTablet state changed. Reason: New leader a761a21430de4522ab92832bf038148e. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.716002 5128 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.717537 5126 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.717656 5126 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:18.717895 5148 mvcc.cc:205] Tried to move back new op lower bound from 7298998348633010176 to 7298998348486860800. Current Snapshot: MvccSnapshot[applied={T|T < 7298998348633010176}]
I20260620 19:00:18.718184 5127 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.718277 5127 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P b81e06e40f7d40b7af4cc98bd0a176d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.723695 5126 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.723799 5126 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P a761a21430de4522ab92832bf038148e: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:18.723556 5150 mvcc.cc:205] Tried to move back new op lower bound from 7298998348633010176 to 7298998348486860800. Current Snapshot: MvccSnapshot[applied={T|T < 7298998348633010176 or (T in {7298998348633010176})}]
I20260620 19:00:18.724819 5128 tablet_replica.cc:439] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } } }
I20260620 19:00:18.724887 5149 mvcc.cc:205] Tried to move back new op lower bound from 7298998348633010176 to 7298998348486860800. Current Snapshot: MvccSnapshot[applied={T|T < 7298998348633010176}]
I20260620 19:00:18.724928 5128 tablet_replica.cc:442] T 0d9923fd19a4411b8cf90910c94d5c37 P 7be750056bc44b3a81394a9663938701: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:18.730630 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.726210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.2.208.131:44815)
0620 19:00:18.726317 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.726321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.726367 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:00:18.726913 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.726916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.726973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.727083 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:18.727517 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.728298 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.729125 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.729477 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.729581 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.729789 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.730254 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.730375 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.730449 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.730501 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:00:18.740957 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.737308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:57451 (local address 127.2.208.129:33127)
0620 19:00:18.737421 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.737425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.737776 (+ 351us) server_negotiation.cc:407] Connection header received
0620 19:00:18.737872 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.737876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.737960 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.738060 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:18.738353 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.738858 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.740025 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.740232 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.740306 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.740601 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.740658 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.740709 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.740824 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.740864 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:18.741032 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.737204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.2.208.131:57451)
0620 19:00:18.737660 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.737674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.737769 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.738084 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.738087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.738094 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:18.738235 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.738241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.738957 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.738960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.739916 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.739926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.740333 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.740337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.740417 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.740460 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.740738 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.740744 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.740808 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.740879 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":117,"threads_started":1}
I20260620 19:00:18.747560 4898 raft_consensus.cc:1261] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Refusing update from remote peer b81e06e40f7d40b7af4cc98bd0a176d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:18.747679 4771 raft_consensus.cc:1261] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Refusing update from remote peer b81e06e40f7d40b7af4cc98bd0a176d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:18.748011 5130 consensus_queue.cc:1035] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:18.748168 5127 consensus_queue.cc:1035] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:18.776140 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.770556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:44925 (local address 127.2.208.131:44815)
0620 19:00:18.770651 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:18.770655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:18.771675 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 19:00:18.771794 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:18.771797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:18.771842 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:18.771917 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:18.772917 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.773688 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.774902 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.775264 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.775352 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.775708 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.775782 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.775821 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:18.775958 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:00:18.776006 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:00:18.776515 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:18.770287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44815 (local address 127.2.208.129:44925)
0620 19:00:18.771582 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:18.771597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:18.771699 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:18.771946 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:18.771949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:18.771957 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:18.772800 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.772808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.773819 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.773822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.774778 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:18.774788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:18.775440 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:18.775444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:18.775527 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:18.775574 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:18.775875 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:18.775880 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:00:18.776321 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 19:00:18.776397 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":80,"threads_started":1}
I20260620 19:00:18.800967 4614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:18.801168 4614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:18.803750 4614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Sending DeleteTablet for 3 replicas of tablet 54ca0881554742eeabb3024d1fe070e7
I20260620 19:00:18.804471 4751 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ca0881554742eeabb3024d1fe070e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:18 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:45174
I20260620 19:00:18.805001 4614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:00:18.805137 5031 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ca0881554742eeabb3024d1fe070e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:18 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47822
W20260620 19:00:18.805389 4614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:18.805874 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ca0881554742eeabb3024d1fe070e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:18 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39538
I20260620 19:00:18.806401 5170 tablet_replica.cc:330] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: stopping tablet replica
I20260620 19:00:18.806594 5171 tablet_replica.cc:330] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: stopping tablet replica
I20260620 19:00:18.806932 5171 raft_consensus.cc:2229] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:18.808364 5169 tablet_replica.cc:330] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: stopping tablet replica
I20260620 19:00:18.808607 5169 raft_consensus.cc:2229] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:18.808751 5169 raft_consensus.cc:2258] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:18.810508 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 8152294a6d01463faa9e65d90afb16c5 (DEFAULT_TABLE table=test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.810705 5169 ts_tablet_manager.cc:1905] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:18.810782 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152294a6d01463faa9e65d90afb16c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.813902 5120 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Bootstrap starting.
I20260620 19:00:18.814334 5169 ts_tablet_manager.cc:1918] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:18.815064 5169 log.cc:1196] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/54ca0881554742eeabb3024d1fe070e7
I20260620 19:00:18.815320 5120 tablet_bootstrap.cc:654] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.815465 5169 ts_tablet_manager.cc:1939] T 54ca0881554742eeabb3024d1fe070e7 P a761a21430de4522ab92832bf038148e: Deleting consensus metadata
I20260620 19:00:18.807247 5170 raft_consensus.cc:2229] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:18.811514 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 8152294a6d01463faa9e65d90afb16c5 (DEFAULT_TABLE table=test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.816453 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152294a6d01463faa9e65d90afb16c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.816490 4598 catalog_manager.cc:4841] TS a761a21430de4522ab92832bf038148e (127.2.208.129:33127): tablet 54ca0881554742eeabb3024d1fe070e7 (table test-table [id=8e6e97f323e743178b19d4f2cb47b425]) successfully deleted
I20260620 19:00:18.817237 5120 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: No bootstrap required, opened a new log
I20260620 19:00:18.817301 5120 ts_tablet_manager.cc:1397] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:18.817690 5120 raft_consensus.cc:348] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.817783 5120 raft_consensus.cc:374] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.817812 5120 raft_consensus.cc:729] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a761a21430de4522ab92832bf038148e, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.817914 5120 consensus_queue.cc:260] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.818884 5121 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Bootstrap starting.
I20260620 19:00:18.819192 5120 ts_tablet_manager.cc:1428] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:18.819789 5121 tablet_bootstrap.cc:654] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.822155 5121 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: No bootstrap required, opened a new log
I20260620 19:00:18.822235 5121 ts_tablet_manager.cc:1397] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:00:18.822635 5121 raft_consensus.cc:348] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.822741 5121 raft_consensus.cc:374] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.822781 5121 raft_consensus.cc:729] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.822892 5121 consensus_queue.cc:260] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.823143 5121 ts_tablet_manager.cc:1428] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:18.824162 5170 raft_consensus.cc:2258] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:18.811229 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 8152294a6d01463faa9e65d90afb16c5 (DEFAULT_TABLE table=test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:18.824477 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152294a6d01463faa9e65d90afb16c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:18.826880 5122 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Bootstrap starting.
I20260620 19:00:18.827389 5171 raft_consensus.cc:2258] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:18.828147 5122 tablet_bootstrap.cc:654] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:18.829357 5170 ts_tablet_manager.cc:1905] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:18.830109 5171 ts_tablet_manager.cc:1905] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:18.832041 5122 tablet_bootstrap.cc:492] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: No bootstrap required, opened a new log
I20260620 19:00:18.832120 5122 ts_tablet_manager.cc:1397] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:00:18.832561 5122 raft_consensus.cc:348] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.832674 5122 raft_consensus.cc:374] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:18.832710 5122 raft_consensus.cc:729] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be750056bc44b3a81394a9663938701, State: Initialized, Role: FOLLOWER
I20260620 19:00:18.832828 5122 consensus_queue.cc:260] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:18.833089 5122 ts_tablet_manager.cc:1428] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:18.835058 5170 ts_tablet_manager.cc:1918] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:18.835947 5171 ts_tablet_manager.cc:1918] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:18.836089 5171 log.cc:1196] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/54ca0881554742eeabb3024d1fe070e7
I20260620 19:00:18.836407 5171 ts_tablet_manager.cc:1939] T 54ca0881554742eeabb3024d1fe070e7 P 7be750056bc44b3a81394a9663938701: Deleting consensus metadata
I20260620 19:00:18.836506 5170 log.cc:1196] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/54ca0881554742eeabb3024d1fe070e7
I20260620 19:00:18.836827 5170 ts_tablet_manager.cc:1939] T 54ca0881554742eeabb3024d1fe070e7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting consensus metadata
I20260620 19:00:18.837896 4598 catalog_manager.cc:4841] TS 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813): tablet 54ca0881554742eeabb3024d1fe070e7 (table test-table [id=8e6e97f323e743178b19d4f2cb47b425]) successfully deleted
I20260620 19:00:18.839008 4598 catalog_manager.cc:4841] TS b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815): tablet 54ca0881554742eeabb3024d1fe070e7 (table test-table [id=8e6e97f323e743178b19d4f2cb47b425]) successfully deleted
I20260620 19:00:19.202605 5167 raft_consensus.cc:482] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:19.202729 5167 raft_consensus.cc:504] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:19.203188 5167 leader_election.cc:288] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:19.203688 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152294a6d01463faa9e65d90afb16c5" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" is_pre_election: true
I20260620 19:00:19.203713 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152294a6d01463faa9e65d90afb16c5" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701" is_pre_election: true
I20260620 19:00:19.203845 5051 raft_consensus.cc:2454] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:19.203860 4898 raft_consensus.cc:2454] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:19.204157 4702 leader_election.cc:302] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, a761a21430de4522ab92832bf038148e; no voters:
I20260620 19:00:19.204334 5167 raft_consensus.cc:2781] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:19.204388 5167 raft_consensus.cc:482] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:19.204427 5167 raft_consensus.cc:3037] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:19.205406 5167 raft_consensus.cc:504] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:19.205744 5167 leader_election.cc:288] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Requested vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:19.206012 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152294a6d01463faa9e65d90afb16c5" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701"
I20260620 19:00:19.206120 4898 raft_consensus.cc:3037] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:19.206118 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152294a6d01463faa9e65d90afb16c5" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
I20260620 19:00:19.206243 5051 raft_consensus.cc:3037] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:19.207069 5051 raft_consensus.cc:2454] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:19.207131 4898 raft_consensus.cc:2454] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:19.207440 4702 leader_election.cc:302] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a761a21430de4522ab92832bf038148e, b81e06e40f7d40b7af4cc98bd0a176d8; no voters:
I20260620 19:00:19.207605 5167 raft_consensus.cc:2781] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:19.207686 5167 raft_consensus.cc:686] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 1 LEADER]: Becoming Leader. State: Replica: a761a21430de4522ab92832bf038148e, State: Running, Role: LEADER
I20260620 19:00:19.207823 5167 consensus_queue.cc:237] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:19.209695 4614 catalog_manager.cc:5495] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e reported cstate change: term changed from 0 to 1, leader changed from <none> to a761a21430de4522ab92832bf038148e (127.2.208.129). New cstate: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:19.271996 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:19.269107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.2.208.129:33127)
0620 19:00:19.269358 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:19.269362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:19.269374 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:19.269455 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:19.269457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:19.269497 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:19.269562 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:19.269763 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.270261 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.271065 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.271273 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.271345 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:19.271631 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:19.271696 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:19.271751 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:19.271848 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:00:19.271885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260620 19:00:19.280085 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:19.277153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.2.208.129:36721)
0620 19:00:19.277339 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:19.277352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:19.277436 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:19.277617 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:19.277619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:19.277625 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:19.277716 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:19.277720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.278355 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.278357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:19.279017 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:19.279026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.279472 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.279475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:19.279552 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:19.279574 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:19.279812 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:19.279819 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:00:19.279884 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:00:19.279956 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":17,"thread_start_us":64,"threads_started":1}
I20260620 19:00:19.280085 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:19.277252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:36721 (local address 127.2.208.129:33127)
0620 19:00:19.277363 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:19.277366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:19.277431 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:00:19.277498 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:19.277500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:19.277536 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:19.277595 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:19.277805 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.278273 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.279128 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:19.279326 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:19.279475 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:19.279669 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:19.279725 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:19.279810 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:19.279891 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:00:19.279936 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":46}
I20260620 19:00:19.285269 4898 raft_consensus.cc:1261] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:19.285307 5051 raft_consensus.cc:1261] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:19.285773 5182 consensus_queue.cc:1035] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:19.285913 5167 consensus_queue.cc:1035] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:00:19.298975 5148 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:00:20.856536 4614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:20.856680 4614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:20.859115 4614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Sending DeleteTablet for 3 replicas of tablet 8152294a6d01463faa9e65d90afb16c5
I20260620 19:00:20.859618 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152294a6d01463faa9e65d90afb16c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:45174
I20260620 19:00:20.859715 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152294a6d01463faa9e65d90afb16c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39538
I20260620 19:00:20.859760 5031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152294a6d01463faa9e65d90afb16c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47822
I20260620 19:00:20.860049 5208 tablet_replica.cc:330] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: stopping tablet replica
I20260620 19:00:20.860230 5208 raft_consensus.cc:2229] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:20.860383 5208 raft_consensus.cc:2258] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.860627 5206 tablet_replica.cc:330] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: stopping tablet replica
I20260620 19:00:20.860767 5206 raft_consensus.cc:2229] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:20.861034 5206 raft_consensus.cc:2258] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.861321 5207 tablet_replica.cc:330] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: stopping tablet replica
I20260620 19:00:20.861755 5208 ts_tablet_manager.cc:1905] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.861755 5206 ts_tablet_manager.cc:1905] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.862732 5207 raft_consensus.cc:2229] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:20.862887 5207 raft_consensus.cc:2258] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.863617 5207 ts_tablet_manager.cc:1905] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.864329 5206 ts_tablet_manager.cc:1918] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:00:20.864329 5208 ts_tablet_manager.cc:1918] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:00:20.864392 5206 log.cc:1196] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/8152294a6d01463faa9e65d90afb16c5
I20260620 19:00:20.864392 5208 log.cc:1196] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/8152294a6d01463faa9e65d90afb16c5
I20260620 19:00:20.864660 5208 ts_tablet_manager.cc:1939] T 8152294a6d01463faa9e65d90afb16c5 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting consensus metadata
I20260620 19:00:20.864666 5206 ts_tablet_manager.cc:1939] T 8152294a6d01463faa9e65d90afb16c5 P a761a21430de4522ab92832bf038148e: Deleting consensus metadata
I20260620 19:00:20.865367 4598 catalog_manager.cc:4841] TS b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815): tablet 8152294a6d01463faa9e65d90afb16c5 (table test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]) successfully deleted
I20260620 19:00:20.865536 4598 catalog_manager.cc:4841] TS a761a21430de4522ab92832bf038148e (127.2.208.129:33127): tablet 8152294a6d01463faa9e65d90afb16c5 (table test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]) successfully deleted
I20260620 19:00:20.866264 5207 ts_tablet_manager.cc:1918] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:00:20.866326 5207 log.cc:1196] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/8152294a6d01463faa9e65d90afb16c5
I20260620 19:00:20.866648 5207 ts_tablet_manager.cc:1939] T 8152294a6d01463faa9e65d90afb16c5 P 7be750056bc44b3a81394a9663938701: Deleting consensus metadata
I20260620 19:00:20.867303 4598 catalog_manager.cc:4841] TS 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813): tablet 8152294a6d01463faa9e65d90afb16c5 (table test-table [id=22ba3fbd5b5c466c9961e6e6a19d6a16]) successfully deleted
Jun 20 19:00:20 dist-test-slave-tsm4 krb5kdc[4569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:00:20 dist-test-slave-tsm4 krb5kdc[4569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:20.887953 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:20.879766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.2.208.190:43167)
0620 19:00:20.880056 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:20.880063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:20.880077 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:20.880188 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:20.880191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:20.880233 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:20.880310 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:20.880950 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:20.881508 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:20.882319 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:20.882553 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:20.882626 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:20.885045 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:20.885048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:20.885051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:20.885073 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:20.886411 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:20.886781 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:20.886784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:20.886785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:20.886857 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:20.887072 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:20.887074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:20.887076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:20.887245 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:20.887344 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:20.887592 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:00:20.887730 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
I20260620 19:00:20.890282 4614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:20.890748 4614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:20.895077 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 68b21389326340ab99f42715f73ca1c7 (DEFAULT_TABLE table=test-table [id=0c703a32e4984c82aa2a2598f71f284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.895221 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 68b21389326340ab99f42715f73ca1c7 (DEFAULT_TABLE table=test-table [id=0c703a32e4984c82aa2a2598f71f284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.895253 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 68b21389326340ab99f42715f73ca1c7 (DEFAULT_TABLE table=test-table [id=0c703a32e4984c82aa2a2598f71f284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.895473 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b21389326340ab99f42715f73ca1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.895537 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b21389326340ab99f42715f73ca1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.895537 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b21389326340ab99f42715f73ca1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.897571 5217 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Bootstrap starting.
I20260620 19:00:20.898191 5219 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Bootstrap starting.
I20260620 19:00:20.898619 5217 tablet_bootstrap.cc:654] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.899240 5219 tablet_bootstrap.cc:654] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.899999 5217 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: No bootstrap required, opened a new log
I20260620 19:00:20.900092 5217 ts_tablet_manager.cc:1397] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:20.900339 5218 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Bootstrap starting.
I20260620 19:00:20.900573 5217 raft_consensus.cc:348] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.900774 5217 raft_consensus.cc:374] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.900852 5217 raft_consensus.cc:729] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a761a21430de4522ab92832bf038148e, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.901010 5217 consensus_queue.cc:260] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.901214 5218 tablet_bootstrap.cc:654] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.901315 5217 ts_tablet_manager.cc:1428] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 19:00:20.902081 5219 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: No bootstrap required, opened a new log
I20260620 19:00:20.902176 5219 ts_tablet_manager.cc:1397] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 19:00:20.902357 5218 tablet_bootstrap.cc:492] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: No bootstrap required, opened a new log
I20260620 19:00:20.902451 5218 ts_tablet_manager.cc:1397] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:20.902623 5219 raft_consensus.cc:348] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.902729 5219 raft_consensus.cc:374] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.902802 5219 raft_consensus.cc:729] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be750056bc44b3a81394a9663938701, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.902832 5218 raft_consensus.cc:348] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.902976 5218 raft_consensus.cc:374] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.902959 5219 consensus_queue.cc:260] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.903057 5218 raft_consensus.cc:729] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.903192 5218 consensus_queue.cc:260] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.903388 5219 ts_tablet_manager.cc:1428] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:20.903625 5218 ts_tablet_manager.cc:1428] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:20.917491 5187 raft_consensus.cc:482] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:20.917634 5187 raft_consensus.cc:504] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.917976 5187 leader_election.cc:288] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:20.918293 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b21389326340ab99f42715f73ca1c7" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701" is_pre_election: true
I20260620 19:00:20.918354 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b21389326340ab99f42715f73ca1c7" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" is_pre_election: true
I20260620 19:00:20.918437 4898 raft_consensus.cc:2454] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:20.918498 5051 raft_consensus.cc:2454] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 0.
I20260620 19:00:20.918761 4702 leader_election.cc:302] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, a761a21430de4522ab92832bf038148e; no voters:
I20260620 19:00:20.918953 5187 raft_consensus.cc:2781] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:20.919016 5187 raft_consensus.cc:482] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:20.919044 5187 raft_consensus.cc:3037] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:20.920174 5187 raft_consensus.cc:504] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.920485 5187 leader_election.cc:288] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Requested vote from peers 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:20.920782 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b21389326340ab99f42715f73ca1c7" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7be750056bc44b3a81394a9663938701"
I20260620 19:00:20.920823 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b21389326340ab99f42715f73ca1c7" candidate_uuid: "a761a21430de4522ab92832bf038148e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
I20260620 19:00:20.920902 4898 raft_consensus.cc:3037] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:20.920948 5051 raft_consensus.cc:3037] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:20.922006 4898 raft_consensus.cc:2454] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:20.922047 5051 raft_consensus.cc:2454] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a761a21430de4522ab92832bf038148e in term 1.
I20260620 19:00:20.922282 4702 leader_election.cc:302] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a761a21430de4522ab92832bf038148e, b81e06e40f7d40b7af4cc98bd0a176d8; no voters:
I20260620 19:00:20.922473 5187 raft_consensus.cc:2781] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:20.922595 5187 raft_consensus.cc:686] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 1 LEADER]: Becoming Leader. State: Replica: a761a21430de4522ab92832bf038148e, State: Running, Role: LEADER
I20260620 19:00:20.922732 5187 consensus_queue.cc:237] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.924234 4611 catalog_manager.cc:5495] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e reported cstate change: term changed from 0 to 1, leader changed from <none> to a761a21430de4522ab92832bf038148e (127.2.208.129). New cstate: current_term: 1 leader_uuid: "a761a21430de4522ab92832bf038148e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:20.940412 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:20.937563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.2.208.129:33127)
0620 19:00:20.937784 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:20.937790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:20.937803 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:20.937840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:20.937842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:20.937880 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:20.937947 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:20.938205 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:20.938698 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:20.939508 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:20.939770 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:20.939847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:20.940077 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:20.940142 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:20.940192 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:20.940263 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:00:20.940301 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260620 19:00:20.946290 5051 raft_consensus.cc:1261] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:20.946311 4898 raft_consensus.cc:1261] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Refusing update from remote peer a761a21430de4522ab92832bf038148e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:20.946707 5226 consensus_queue.cc:1035] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:20.946861 5182 consensus_queue.cc:1035] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:20.972647 4611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:20.972813 4611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:20.974660 4611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Sending DeleteTablet for 3 replicas of tablet 68b21389326340ab99f42715f73ca1c7
I20260620 19:00:20.975234 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b21389326340ab99f42715f73ca1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39538
I20260620 19:00:20.975266 4751 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b21389326340ab99f42715f73ca1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:45174
I20260620 19:00:20.975276 5031 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b21389326340ab99f42715f73ca1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:20 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47822
I20260620 19:00:20.975430 5207 tablet_replica.cc:330] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: stopping tablet replica
I20260620 19:00:20.975430 5206 tablet_replica.cc:330] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: stopping tablet replica
I20260620 19:00:20.975539 5206 raft_consensus.cc:2229] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:20.975540 5207 raft_consensus.cc:2229] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:20.975654 5207 raft_consensus.cc:2258] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.975699 5206 raft_consensus.cc:2258] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.976483 5206 ts_tablet_manager.cc:1905] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.976794 4611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:20.977109 4611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:20.978262 5208 tablet_replica.cc:330] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: stopping tablet replica
I20260620 19:00:20.978375 5208 raft_consensus.cc:2229] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:20.978475 5208 raft_consensus.cc:2258] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:20.979125 5206 ts_tablet_manager.cc:1918] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:20.979183 5206 log.cc:1196] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/68b21389326340ab99f42715f73ca1c7
I20260620 19:00:20.979686 5206 ts_tablet_manager.cc:1939] T 68b21389326340ab99f42715f73ca1c7 P a761a21430de4522ab92832bf038148e: Deleting consensus metadata
I20260620 19:00:20.980298 4598 catalog_manager.cc:4841] TS a761a21430de4522ab92832bf038148e (127.2.208.129:33127): tablet 68b21389326340ab99f42715f73ca1c7 (table test-table [id=0c703a32e4984c82aa2a2598f71f284c]) successfully deleted
I20260620 19:00:20.981555 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 38b205d567d04ab1a31659fc10c6c26a (DEFAULT_TABLE table=test-table [id=796be8b9d67346e49891117fa66f29f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.981701 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 38b205d567d04ab1a31659fc10c6c26a (DEFAULT_TABLE table=test-table [id=796be8b9d67346e49891117fa66f29f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.981807 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b205d567d04ab1a31659fc10c6c26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.981916 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b205d567d04ab1a31659fc10c6c26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.983470 5217 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: Bootstrap starting.
I20260620 19:00:20.983631 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 38b205d567d04ab1a31659fc10c6c26a (DEFAULT_TABLE table=test-table [id=796be8b9d67346e49891117fa66f29f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:20.983850 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b205d567d04ab1a31659fc10c6c26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:20.984257 5217 tablet_bootstrap.cc:654] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.985342 5217 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: No bootstrap required, opened a new log
I20260620 19:00:20.985410 5217 ts_tablet_manager.cc:1397] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:20.985783 5217 raft_consensus.cc:348] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.985891 5217 raft_consensus.cc:374] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.985922 5217 raft_consensus.cc:729] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a761a21430de4522ab92832bf038148e, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.986033 5217 consensus_queue.cc:260] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.986092 5219 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Bootstrap starting.
I20260620 19:00:20.986163 5218 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Bootstrap starting.
I20260620 19:00:20.986281 5217 ts_tablet_manager.cc:1428] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:20.986904 5218 tablet_bootstrap.cc:654] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.986954 5219 tablet_bootstrap.cc:654] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:20.987917 5207 ts_tablet_manager.cc:1905] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.987957 5208 ts_tablet_manager.cc:1905] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:20.990104 5208 ts_tablet_manager.cc:1918] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:20.990104 5207 ts_tablet_manager.cc:1918] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:20.990167 5207 log.cc:1196] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/68b21389326340ab99f42715f73ca1c7
I20260620 19:00:20.990168 5208 log.cc:1196] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/68b21389326340ab99f42715f73ca1c7
I20260620 19:00:20.990414 5207 ts_tablet_manager.cc:1939] T 68b21389326340ab99f42715f73ca1c7 P 7be750056bc44b3a81394a9663938701: Deleting consensus metadata
I20260620 19:00:20.990414 5208 ts_tablet_manager.cc:1939] T 68b21389326340ab99f42715f73ca1c7 P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting consensus metadata
I20260620 19:00:20.991040 4598 catalog_manager.cc:4841] TS b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815): tablet 68b21389326340ab99f42715f73ca1c7 (table test-table [id=0c703a32e4984c82aa2a2598f71f284c]) successfully deleted
I20260620 19:00:20.991170 4598 catalog_manager.cc:4841] TS 7be750056bc44b3a81394a9663938701 (127.2.208.130:37813): tablet 68b21389326340ab99f42715f73ca1c7 (table test-table [id=0c703a32e4984c82aa2a2598f71f284c]) successfully deleted
I20260620 19:00:20.991817 5218 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: No bootstrap required, opened a new log
I20260620 19:00:20.991900 5218 ts_tablet_manager.cc:1397] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:00:20.992237 5218 raft_consensus.cc:348] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.992349 5218 raft_consensus.cc:374] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.992385 5218 raft_consensus.cc:729] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81e06e40f7d40b7af4cc98bd0a176d8, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.992517 5218 consensus_queue.cc:260] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.992776 5218 ts_tablet_manager.cc:1428] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:20.993836 5219 tablet_bootstrap.cc:492] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: No bootstrap required, opened a new log
I20260620 19:00:20.993927 5219 ts_tablet_manager.cc:1397] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 19:00:20.994334 5219 raft_consensus.cc:348] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.994439 5219 raft_consensus.cc:374] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:20.994465 5219 raft_consensus.cc:729] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be750056bc44b3a81394a9663938701, State: Initialized, Role: FOLLOWER
I20260620 19:00:20.994582 5219 consensus_queue.cc:260] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:20.994834 5219 ts_tablet_manager.cc:1428] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:21.241070 5223 raft_consensus.cc:482] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:21.241215 5223 raft_consensus.cc:504] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:21.242249 5223 leader_election.cc:288] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a761a21430de4522ab92832bf038148e (127.2.208.129:33127), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:21.246552 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.242855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44815 (local address 127.2.208.130:40297)
0620 19:00:21.243156 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:21.243172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:21.243275 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:21.243524 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:21.243526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:21.243539 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:21.243685 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.243692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.244402 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.244406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.245374 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.245383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.246087 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.246091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.246203 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.246239 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:00:21.246313 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:21.246396 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260620 19:00:21.246577 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.243013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:40297 (local address 127.2.208.131:44815)
0620 19:00:21.243263 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:21.243269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:21.243284 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:21.243353 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:21.243382 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:21.243425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:21.243508 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:21.243795 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.244304 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.245519 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.245948 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.246037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.246095 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:21.246346 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:00:21.246422 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
I20260620 19:00:21.246799 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.242504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.2.208.130:45123)
0620 19:00:21.242740 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:21.242757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:21.242879 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:21.243328 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:21.243332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:21.243350 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:21.243519 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.243526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.244494 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.244508 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.245367 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.245377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.246338 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.246342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.246447 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.246470 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:00:21.246546 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:00:21.246618 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":92,"thread_start_us":87,"threads_started":1}
I20260620 19:00:21.246812 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.242714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:45123 (local address 127.2.208.129:33127)
0620 19:00:21.242832 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:21.242839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:21.242859 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:00:21.243051 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:21.243055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:21.243115 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:21.243175 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:21.243631 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.244340 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.245574 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.246190 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.246321 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.246365 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:21.246586 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:00:21.246676 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:00:21.247236 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b205d567d04ab1a31659fc10c6c26a" candidate_uuid: "7be750056bc44b3a81394a9663938701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a761a21430de4522ab92832bf038148e" is_pre_election: true
I20260620 19:00:21.247259 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b205d567d04ab1a31659fc10c6c26a" candidate_uuid: "7be750056bc44b3a81394a9663938701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" is_pre_election: true
I20260620 19:00:21.247426 4771 raft_consensus.cc:2454] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7be750056bc44b3a81394a9663938701 in term 0.
I20260620 19:00:21.247450 5051 raft_consensus.cc:2454] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7be750056bc44b3a81394a9663938701 in term 0.
I20260620 19:00:21.247800 4842 leader_election.cc:302] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, a761a21430de4522ab92832bf038148e; no voters:
I20260620 19:00:21.248077 5223 raft_consensus.cc:2781] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:21.248139 5223 raft_consensus.cc:482] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:21.248175 5223 raft_consensus.cc:3037] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:21.249176 5223 raft_consensus.cc:504] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:21.249543 5223 leader_election.cc:288] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [CANDIDATE]: Term 1 election: Requested vote from peers a761a21430de4522ab92832bf038148e (127.2.208.129:33127), b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815)
I20260620 19:00:21.249895 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b205d567d04ab1a31659fc10c6c26a" candidate_uuid: "7be750056bc44b3a81394a9663938701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a761a21430de4522ab92832bf038148e"
I20260620 19:00:21.249895 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b205d567d04ab1a31659fc10c6c26a" candidate_uuid: "7be750056bc44b3a81394a9663938701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8"
I20260620 19:00:21.250016 4771 raft_consensus.cc:3037] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:21.250016 5051 raft_consensus.cc:3037] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:21.251077 4771 raft_consensus.cc:2454] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be750056bc44b3a81394a9663938701 in term 1.
I20260620 19:00:21.251077 5051 raft_consensus.cc:2454] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7be750056bc44b3a81394a9663938701 in term 1.
I20260620 19:00:21.251341 4842 leader_election.cc:302] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7be750056bc44b3a81394a9663938701, a761a21430de4522ab92832bf038148e; no voters:
I20260620 19:00:21.251549 5223 raft_consensus.cc:2781] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:21.251751 5223 raft_consensus.cc:686] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 1 LEADER]: Becoming Leader. State: Replica: 7be750056bc44b3a81394a9663938701, State: Running, Role: LEADER
I20260620 19:00:21.252007 5223 consensus_queue.cc:237] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } }
I20260620 19:00:21.254231 4611 catalog_manager.cc:5495] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7be750056bc44b3a81394a9663938701 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "7be750056bc44b3a81394a9663938701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7be750056bc44b3a81394a9663938701" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:21.288364 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.284608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.2.208.130:37813)
0620 19:00:21.284928 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:21.284934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:21.284945 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:00:21.285008 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:21.285010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:21.285057 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:21.285186 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:21.285421 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.285937 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.286759 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.287005 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.287147 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.287477 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:21.287920 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:21.288051 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:21.288112 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:00:21.288162 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":153,"threads_started":1}
I20260620 19:00:21.298924 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.295195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:35485 (local address 127.2.208.129:33127)
0620 19:00:21.295403 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:21.295408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:21.295562 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:00:21.295724 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:21.295727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:21.295770 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:21.295832 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:21.296396 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.296897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.297822 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.298025 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.298099 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.298417 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:21.298493 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:21.298584 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:21.298724 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:00:21.298774 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 19:00:21.298944 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.295097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.2.208.130:35485)
0620 19:00:21.295482 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:21.295497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:21.295624 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:21.295862 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:21.295865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:21.295872 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:21.296278 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.296284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.297001 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.297003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.297731 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.297740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.298128 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.298131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.298204 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.298246 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:21.298600 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:21.298609 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:00:21.298699 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:00:21.298779 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260620 19:00:21.305039 5051 raft_consensus.cc:1261] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Refusing update from remote peer 7be750056bc44b3a81394a9663938701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:21.305039 4771 raft_consensus.cc:1261] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e [term 1 FOLLOWER]: Refusing update from remote peer 7be750056bc44b3a81394a9663938701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:21.305567 5238 consensus_queue.cc:1035] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a761a21430de4522ab92832bf038148e" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:21.305763 5223 consensus_queue.cc:1035] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81e06e40f7d40b7af4cc98bd0a176d8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:21.325404 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.321976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:33405 (local address 127.2.208.130:37813)
0620 19:00:21.322100 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:21.322104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:21.322191 (+ 87us) server_negotiation.cc:407] Connection header received
0620 19:00:21.322346 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:21.322349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:21.322400 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:21.322493 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:21.322657 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.323154 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.324388 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.324723 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.324821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.325068 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:21.325131 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:21.325187 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:21.325261 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:00:21.325302 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:00:21.325433 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:21.321848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37813 (local address 127.2.208.129:33405)
0620 19:00:21.322117 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:21.322133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:21.322243 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:21.322503 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:21.322505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:21.322511 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:21.322597 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.322601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.323283 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.323286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.324275 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:21.324283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:21.324841 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:21.324846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:21.324932 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:21.324960 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:21.325203 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:21.325208 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:00:21.325255 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:00:21.325319 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260620 19:00:21.326103 5190 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:00:22.881325 4614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:22.881525 4614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:22.883841 4614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 779cc85e0ee64783904e6a495a0a1a11: Sending DeleteTablet for 3 replicas of tablet 38b205d567d04ab1a31659fc10c6c26a
I20260620 19:00:22.884346 4751 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b205d567d04ab1a31659fc10c6c26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:22 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:45174
I20260620 19:00:22.884392 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b205d567d04ab1a31659fc10c6c26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:22 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39538
I20260620 19:00:22.884478 5031 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b205d567d04ab1a31659fc10c6c26a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:22 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47822
I20260620 19:00:22.884768 5274 tablet_replica.cc:330] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: stopping tablet replica
I20260620 19:00:22.884766 5273 tablet_replica.cc:330] T 38b205d567d04ab1a31659fc10c6c26a P a761a21430de4522ab92832bf038148e: stopping tablet replica
I20260620 19:00:22.884770 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4686
I20260620 19:00:22.886243 5274 raft_consensus.cc:2229] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:22.886375 5274 raft_consensus.cc:2258] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:22.886598 5272 tablet_replica.cc:330] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: stopping tablet replica
I20260620 19:00:22.886718 5272 raft_consensus.cc:2229] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:22.887072 5274 ts_tablet_manager.cc:1905] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:22.887108 5272 raft_consensus.cc:2258] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:22.889159 5274 ts_tablet_manager.cc:1918] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:00:22.889222 5274 log.cc:1196] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/38b205d567d04ab1a31659fc10c6c26a
I20260620 19:00:22.889546 5274 ts_tablet_manager.cc:1939] T 38b205d567d04ab1a31659fc10c6c26a P b81e06e40f7d40b7af4cc98bd0a176d8: Deleting consensus metadata
I20260620 19:00:22.889933 5272 ts_tablet_manager.cc:1905] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:22.890199 4598 catalog_manager.cc:4841] TS b81e06e40f7d40b7af4cc98bd0a176d8 (127.2.208.131:44815): tablet 38b205d567d04ab1a31659fc10c6c26a (table test-table [id=796be8b9d67346e49891117fa66f29f0]) successfully deleted
I20260620 19:00:22.893252 5272 ts_tablet_manager.cc:1918] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:00:22.893338 5272 log.cc:1196] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/38b205d567d04ab1a31659fc10c6c26a
I20260620 19:00:22.893625 5272 ts_tablet_manager.cc:1939] T 38b205d567d04ab1a31659fc10c6c26a P 7be750056bc44b3a81394a9663938701: Deleting consensus metadata
W20260620 19:00:22.893608 4598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:33127 (error 108)
I20260620 19:00:22.893847 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4826
I20260620 19:00:22.896513 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:22.896126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33127 (local address 127.0.0.1:60110)
0620 19:00:22.896358 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:22.896422 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
W20260620 19:00:22.896641 4598 catalog_manager.cc:4584] TS a761a21430de4522ab92832bf038148e (127.2.208.129:33127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38b205d567d04ab1a31659fc10c6c26a: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33127: connect: Connection refused (error 111)
I20260620 19:00:22.900626 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4966
I20260620 19:00:22.901433 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:22.900637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37813 (local address 127.0.0.1:35596)
0620 19:00:22.901279 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:22.901343 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.130:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 19:00:22.907066 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 4582
2026-06-20T19:00:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6742 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:22 dist-test-slave-tsm4 krb5kdc[5279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:22 dist-test-slave-tsm4 krb5kdc[5279](info): set up 2 sockets
Jun 20 19:00:22 dist-test-slave-tsm4 krb5kdc[5279](info): commencing operation
krb5kdc: starting...
W20260620 19:00:23.951351 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:24.069854 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44783
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:24.177688 5292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:24.177961 5292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:24.178001 5292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:24.182197 5292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:24.182277 5292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:24.182299 5292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:24.182384 5292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:24.182425 5292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:24.188422 5292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:24.189558 5292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:24.190613 5292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:24.199527 5297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.199600 5292 server_base.cc:1034] running on GCE node
W20260620 19:00:24.199537 5298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:24.199884 5300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.200613 5292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:24.201727 5292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:24.202935 5292 hybrid_clock.cc:648] HybridClock initialized: now 1781982024202901 us; error 50 us; skew 500 ppm
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:24.205684 5292 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:24.206585 5292 webserver.cc:460] Webserver started at http://127.2.208.190:36251/ using document root <none> and password file <none>
I20260620 19:00:24.207144 5292 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:24.207237 5292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:24.207515 5292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:24.209622 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "33e9e65fe19c446ea7c047ed79acaf4a"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "292d61b9253a35002b223626f57736ba"
server_key_iv: "b78f55f1ac7701433316ff71e22424ae"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.210204 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "33e9e65fe19c446ea7c047ed79acaf4a"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "292d61b9253a35002b223626f57736ba"
server_key_iv: "b78f55f1ac7701433316ff71e22424ae"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.213999 5292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:00:24.216837 5307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.218019 5292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:24.218156 5292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "33e9e65fe19c446ea7c047ed79acaf4a"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "292d61b9253a35002b223626f57736ba"
server_key_iv: "b78f55f1ac7701433316ff71e22424ae"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.218271 5292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:24.230053 5292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:24.233299 5292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:24.233503 5292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:24.240366 5292 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44783
I20260620 19:00:24.240389 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44783 every 8 connection(s)
I20260620 19:00:24.241262 5292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:24.243840 5360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:24.245831 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5292
I20260620 19:00:24.245975 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:24.246233 2882 external_mini_cluster.cc:1428] Setting key 03074b930f101f2a01081c0cdf5d1c90
I20260620 19:00:24.249544 5360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a: Bootstrap starting.
I20260620 19:00:24.251680 5360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:24.252401 5360 log.cc:824] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:24.254689 5360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a: No bootstrap required, opened a new log
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:24.258809 5360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } }
I20260620 19:00:24.259068 5360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:24.259114 5360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e9e65fe19c446ea7c047ed79acaf4a, State: Initialized, Role: FOLLOWER
I20260620 19:00:24.259622 5360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } }
I20260620 19:00:24.259871 5360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:24.259949 5360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:24.260043 5360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:24.260902 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.247620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.2.208.190:44783)
0620 19:00:24.248021 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:24.248035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:24.248068 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:00:24.248695 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:24.248710 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:24.249020 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:24.249369 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:24.250120 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.251097 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.252018 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.252504 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.252669 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.255465 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:24.255471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:24.255487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:24.255516 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:24.257884 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.258302 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.258308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.258314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.258444 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.258683 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.258686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.258688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.259183 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:24.259392 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:24.259668 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:00:24.259978 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260620 19:00:24.261045 5360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } }
I20260620 19:00:24.261380 5360 leader_election.cc:302] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e9e65fe19c446ea7c047ed79acaf4a; no voters:
I20260620 19:00:24.261683 5360 leader_election.cc:288] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:24.261822 5365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:24.262081 5365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [term 1 LEADER]: Becoming Leader. State: Replica: 33e9e65fe19c446ea7c047ed79acaf4a, State: Running, Role: LEADER
I20260620 19:00:24.262409 5365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } }
I20260620 19:00:24.262892 5360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:24.264046 5366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } } }
I20260620 19:00:24.264173 5366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:24.264573 5374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:24.264541 5367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e9e65fe19c446ea7c047ed79acaf4a. Latest consensus state: current_term: 1 leader_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e9e65fe19c446ea7c047ed79acaf4a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44783 } } }
I20260620 19:00:24.265398 5367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:24.267685 5374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:24.273072 5374 catalog_manager.cc:1347] Generated new cluster ID: 2172131f25d74404a45fbb2356fab1e2
I20260620 19:00:24.273142 5374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:24.286000 5374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:24.286960 5374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:24.296234 5374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e9e65fe19c446ea7c047ed79acaf4a: Generated new TSK 0
I20260620 19:00:24.296896 5374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:24.334434 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:24.456295 5386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:24.456560 5386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:24.456604 5386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:24.460536 5386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:24.460590 5386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:24.460646 5386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:24.465824 5386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:24.466811 5386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:24.467869 5386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:00:24.477504 5386 server_base.cc:1034] running on GCE node
W20260620 19:00:24.477451 5394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:24.477466 5392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:24.477433 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.478092 5386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:24.478667 5386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:24.479831 5386 hybrid_clock.cc:648] HybridClock initialized: now 1781982024479813 us; error 30 us; skew 500 ppm
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:24.482625 5386 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:24.483546 5386 webserver.cc:460] Webserver started at http://127.2.208.129:39153/ using document root <none> and password file <none>
I20260620 19:00:24.484093 5386 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:24.484153 5386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:24.484382 5386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:24.487112 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "99ed4f1c9e2e4e39aa9e980437837131"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "3dd13e94792a1c846f9e2c17b27e7951"
server_key_iv: "3aea2801c68906221ca5807816a2c4b1"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.487869 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "99ed4f1c9e2e4e39aa9e980437837131"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "3dd13e94792a1c846f9e2c17b27e7951"
server_key_iv: "3aea2801c68906221ca5807816a2c4b1"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.491748 5386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:00:24.494112 5401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.495327 5386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:24.495522 5386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "99ed4f1c9e2e4e39aa9e980437837131"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "3dd13e94792a1c846f9e2c17b27e7951"
server_key_iv: "3aea2801c68906221ca5807816a2c4b1"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.495638 5386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:24.517210 5386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:24.520107 5386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:24.520318 5386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:24.520963 5386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:24.521906 5386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:24.521977 5386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.522042 5386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:24.522094 5386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.531141 5386 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:44133
I20260620 19:00:24.531162 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:44133 every 8 connection(s)
I20260620 19:00:24.532076 5386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:24.541330 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5386
I20260620 19:00:24.541419 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:24.541721 2882 external_mini_cluster.cc:1428] Setting key 17fb14be530036ae45b4063d9854537b
I20260620 19:00:24.545867 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.533868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:42929 (local address 127.2.208.190:44783)
0620 19:00:24.534021 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:24.534025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:24.534776 (+ 751us) server_negotiation.cc:407] Connection header received
0620 19:00:24.536245 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:24.536249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:24.536320 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:24.536425 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:24.537607 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.538145 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.539278 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.539584 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.539699 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.542106 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:24.542109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:24.542112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:24.542139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:24.543787 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.544273 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.544280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.544284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.544374 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.544689 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.544695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.544700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.544944 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:24.545047 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:24.545653 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 19:00:24.545752 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:24.546833 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.534192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44783 (local address 127.2.208.129:42929)
0620 19:00:24.534625 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:24.534669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:24.536022 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:24.536609 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:24.536616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:24.536980 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:24.537439 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.537452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.538272 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.538275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.539146 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.539155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.539737 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.539741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.539908 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.540432 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:24.540448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:24.541937 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:24.543913 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.543918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.543929 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.544172 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.544466 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.544468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.544469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.544594 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.545067 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:24.545073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:24.545395 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 19:00:24.545617 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 19:00:24.546040 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
I20260620 19:00:24.548007 5515 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44783
I20260620 19:00:24.548265 5515 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:24.548871 5515 heartbeater.cc:511] Master 127.2.208.190:44783 requested a full tablet report, sending...
I20260620 19:00:24.550443 5324 ts_manager.cc:194] Registered new tserver with Master: 99ed4f1c9e2e4e39aa9e980437837131 (127.2.208.129:44133)
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:24.551635 5324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:42929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:24.570207 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:24.681414 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:24.681653 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:24.681694 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:24.685709 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:24.685760 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:24.685814 5520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:24.690930 5520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:24.692067 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:24.693045 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:24.702497 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:24.702497 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.702662 5520 server_base.cc:1034] running on GCE node
W20260620 19:00:24.702836 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.703259 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:24.703892 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:24.705098 5520 hybrid_clock.cc:648] HybridClock initialized: now 1781982024705079 us; error 30 us; skew 500 ppm
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:24.707792 5520 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:24.708600 5520 webserver.cc:460] Webserver started at http://127.2.208.130:43001/ using document root <none> and password file <none>
I20260620 19:00:24.709161 5520 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:24.709275 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:24.709520 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:24.711537 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "3fff516b582844c9a5072035ac5c4902"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "9dabaabbb73599251cfef67196e8bd17"
server_key_iv: "e76b7736e884eafd829f367cc92d92f3"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.712100 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "3fff516b582844c9a5072035ac5c4902"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "9dabaabbb73599251cfef67196e8bd17"
server_key_iv: "e76b7736e884eafd829f367cc92d92f3"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.715746 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:24.718073 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.719225 5520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:24.719413 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "3fff516b582844c9a5072035ac5c4902"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "9dabaabbb73599251cfef67196e8bd17"
server_key_iv: "e76b7736e884eafd829f367cc92d92f3"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.719532 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:24.736317 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:24.739621 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:24.739830 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:24.740406 5520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:24.741199 5520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:24.741276 5520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.741340 5520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:24.741405 5520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.749681 5520 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:45847
I20260620 19:00:24.749701 5648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:45847 every 8 connection(s)
I20260620 19:00:24.750545 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:24.756300 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5520
I20260620 19:00:24.756522 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:24.756793 2882 external_mini_cluster.cc:1428] Setting key b78180919d1fb30f36d4dc5bbcc2973d
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:24.765034 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.752308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:52319 (local address 127.2.208.190:44783)
0620 19:00:24.752436 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:24.752439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:24.753166 (+ 727us) server_negotiation.cc:407] Connection header received
0620 19:00:24.754568 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:24.754571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:24.754626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:24.754733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:24.755929 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.756452 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.757776 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.758044 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.758148 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.761124 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:24.761127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:24.761128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:24.761151 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:24.762633 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.763203 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.763208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.763210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.763289 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.763686 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.763692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.763696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.764035 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:24.764206 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:24.764721 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 19:00:24.764843 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:24.765813 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.752619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44783 (local address 127.2.208.130:52319)
0620 19:00:24.753024 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:24.753063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:24.754351 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:24.754896 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:24.754903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:24.755251 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:24.755786 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.755799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.756618 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.756621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.757652 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.757662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.758213 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.758217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.758405 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.759010 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:24.759036 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:24.760947 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:24.762748 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.762754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.762769 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.763099 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.763398 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.763403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.763408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.763549 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.764215 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:24.764220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:24.764444 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 19:00:24.764607 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 19:00:24.764847 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:00:24.767061 5649 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44783
I20260620 19:00:24.767277 5649 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:24.767772 5649 heartbeater.cc:511] Master 127.2.208.190:44783 requested a full tablet report, sending...
I20260620 19:00:24.768831 5324 ts_manager.cc:194] Registered new tserver with Master: 3fff516b582844c9a5072035ac5c4902 (127.2.208.130:45847)
I20260620 19:00:24.769467 5324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:52319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:24.786279 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:24.894532 5654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:24.894840 5654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:24.894913 5654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:24.899194 5654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:24.899281 5654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:24.899433 5654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:24.904700 5654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:24.905735 5654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:24.906755 5654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:24.916427 5659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.916589 5654 server_base.cc:1034] running on GCE node
W20260620 19:00:24.916446 5660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:24.916432 5662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:24.917089 5654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:24.917723 5654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:24.918918 5654 hybrid_clock.cc:648] HybridClock initialized: now 1781982024918892 us; error 45 us; skew 500 ppm
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:24.921739 5654 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:24.922727 5654 webserver.cc:460] Webserver started at http://127.2.208.131:46239/ using document root <none> and password file <none>
I20260620 19:00:24.923287 5654 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:24.923419 5654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:24.923615 5654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:24.925609 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "fb0cefa77e694ddf95ed319215f1187d"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "460c29e75e7bb7e473825cb9fa304be7"
server_key_iv: "30830fe126d7065e0226cc3cb5eca362"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.926122 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "fb0cefa77e694ddf95ed319215f1187d"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "460c29e75e7bb7e473825cb9fa304be7"
server_key_iv: "30830fe126d7065e0226cc3cb5eca362"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.929695 5654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 19:00:24.932034 5669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.933238 5654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 19:00:24.933379 5654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "fb0cefa77e694ddf95ed319215f1187d"
format_stamp: "Formatted at 2026-06-20 19:00:24 on dist-test-slave-tsm4"
server_key: "460c29e75e7bb7e473825cb9fa304be7"
server_key_iv: "30830fe126d7065e0226cc3cb5eca362"
server_key_version: "encryptionkey@0"
I20260620 19:00:24.933506 5654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:24.952047 5654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:24.954778 5654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:24.954990 5654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:24.955663 5654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:24.956485 5654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:24.956558 5654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.956625 5654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:24.956678 5654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:24.965494 5654 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:37687
I20260620 19:00:24.965515 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:37687 every 8 connection(s)
I20260620 19:00:24.966390 5654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:24.972206 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5654
I20260620 19:00:24.972314 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:24.972584 2882 external_mini_cluster.cc:1428] Setting key 6c2603cd74519dce59a87693d01a61cd
Jun 20 19:00:24 dist-test-slave-tsm4 krb5kdc[5279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:24.980332 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.968254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39481 (local address 127.2.208.190:44783)
0620 19:00:24.968395 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:24.968400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:24.969194 (+ 794us) server_negotiation.cc:407] Connection header received
0620 19:00:24.970605 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:24.970608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:24.970655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:24.970728 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:24.971829 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.972603 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.974220 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.974486 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.974581 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.977047 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:24.977050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:24.977052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:24.977074 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:24.978298 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.978799 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.978803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.978804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.978880 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:24.979218 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:24.979223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:24.979225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:24.979469 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:24.979575 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:24.980051 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 19:00:24.980159 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:00:24.981125 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:24.968627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44783 (local address 127.2.208.131:39481)
0620 19:00:24.969042 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:24.969083 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:24.970310 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:24.970859 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:24.970866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:24.971204 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:24.971706 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.971719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.972745 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.972748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.974105 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:24.974115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:24.974676 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:24.974680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:24.974828 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:24.975408 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:24.975432 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:24.976841 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:24.978414 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.978419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.978434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.978688 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.978984 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:24.978987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:24.978989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:24.979125 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:24.979571 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:24.979578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:24.979822 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 19:00:24.980015 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 19:00:24.980332 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260620 19:00:24.982172 5783 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44783
I20260620 19:00:24.982391 5783 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:24.982812 5783 heartbeater.cc:511] Master 127.2.208.190:44783 requested a full tablet report, sending...
I20260620 19:00:24.983768 5324 ts_manager.cc:194] Registered new tserver with Master: fb0cefa77e694ddf95ed319215f1187d (127.2.208.131:37687)
I20260620 19:00:24.984270 5324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:39481
I20260620 19:00:24.986297 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:00:24.999128 5363 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:24.997734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.2.208.190:44783)
0620 19:00:24.997914 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:24.997918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:24.997996 (+ 78us) server_negotiation.cc:407] Connection header received
0620 19:00:24.998167 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:24.998169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:24.998212 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:24.998281 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:24.998989 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56632: BlockingRecv error: recv got EOF from 127.0.0.1:56632 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 19:00:25.000164 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5386
I20260620 19:00:25.005774 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5520
I20260620 19:00:25.010972 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5654
I20260620 19:00:25.016290 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5292
2026-06-20T19:00:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2107 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:25 dist-test-slave-tsm4 krb5kdc[5795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:25 dist-test-slave-tsm4 krb5kdc[5795](info): set up 2 sockets
Jun 20 19:00:25 dist-test-slave-tsm4 krb5kdc[5795](info): commencing operation
krb5kdc: starting...
W20260620 19:00:26.055477 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:26.173736 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45359
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:26.285049 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:26.285319 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:26.285359 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:26.289428 5808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:26.289481 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:26.289501 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:26.289543 5808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:26.289578 5808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:26.294724 5808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:45359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:26.295768 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:26.296768 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:26.305159 5814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:26.305182 5816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:26.305159 5813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:26.305430 5808 server_base.cc:1034] running on GCE node
I20260620 19:00:26.306018 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:26.307081 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:26.308259 5808 hybrid_clock.cc:648] HybridClock initialized: now 1781982026308230 us; error 48 us; skew 500 ppm
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:26.316927 5808 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:26.317838 5808 webserver.cc:460] Webserver started at http://127.2.208.190:40055/ using document root <none> and password file <none>
I20260620 19:00:26.318370 5808 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:26.318427 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:26.318603 5808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:26.320672 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "2283009b178044279468c7952c87c47e"
format_stamp: "Formatted at 2026-06-20 19:00:26 on dist-test-slave-tsm4"
server_key: "cc9204b51a9e7ad63e01676484a3c99d"
server_key_iv: "3d69fa7c2565cef792386e03859b76fb"
server_key_version: "encryptionkey@0"
I20260620 19:00:26.321167 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "2283009b178044279468c7952c87c47e"
format_stamp: "Formatted at 2026-06-20 19:00:26 on dist-test-slave-tsm4"
server_key: "cc9204b51a9e7ad63e01676484a3c99d"
server_key_iv: "3d69fa7c2565cef792386e03859b76fb"
server_key_version: "encryptionkey@0"
I20260620 19:00:26.324878 5808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 19:00:26.327145 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:26.328357 5808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:00:26.328474 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2283009b178044279468c7952c87c47e"
format_stamp: "Formatted at 2026-06-20 19:00:26 on dist-test-slave-tsm4"
server_key: "cc9204b51a9e7ad63e01676484a3c99d"
server_key_iv: "3d69fa7c2565cef792386e03859b76fb"
server_key_version: "encryptionkey@0"
I20260620 19:00:26.328560 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:26.343676 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:26.346282 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:26.346427 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:26.353103 5808 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:45359
I20260620 19:00:26.353374 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:45359 every 8 connection(s)
I20260620 19:00:26.354054 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:26.356760 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:26.359964 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5808
I20260620 19:00:26.360076 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:26.360324 2882 external_mini_cluster.cc:1428] Setting key e6b82e9f30b450fc142b4d4eae89e3b7
I20260620 19:00:26.362672 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e: Bootstrap starting.
I20260620 19:00:26.365316 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:26.366155 5876 log.cc:824] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:26.368314 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e: No bootstrap required, opened a new log
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:26.371897 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } }
I20260620 19:00:26.372126 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:26.372205 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2283009b178044279468c7952c87c47e, State: Initialized, Role: FOLLOWER
I20260620 19:00:26.372671 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } }
I20260620 19:00:26.372843 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:26.372910 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:26.373000 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:26.374137 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } }
I20260620 19:00:26.374482 5876 leader_election.cc:302] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2283009b178044279468c7952c87c47e; no voters:
I20260620 19:00:26.374811 5876 leader_election.cc:288] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:26.374903 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:26.361622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.2.208.190:45359)
0620 19:00:26.362021 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:26.362038 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:26.362096 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:00:26.362913 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:26.362936 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:26.363339 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:26.363722 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:26.364388 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:26.365853 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:26.366722 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:26.367242 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:26.367416 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:26.370043 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:26.370048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:26.370061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:26.370086 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:26.371779 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:26.372270 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:26.372275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:26.372295 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:26.372430 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:26.372717 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:26.372719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:26.372720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:26.373204 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:26.373358 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:26.373670 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:00:26.373987 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260620 19:00:26.374912 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:26.375506 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [term 1 LEADER]: Becoming Leader. State: Replica: 2283009b178044279468c7952c87c47e, State: Running, Role: LEADER
I20260620 19:00:26.375797 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } }
I20260620 19:00:26.376060 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:26.376888 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2283009b178044279468c7952c87c47e. Latest consensus state: current_term: 1 leader_uuid: "2283009b178044279468c7952c87c47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } } }
I20260620 19:00:26.377085 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:26.377434 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2283009b178044279468c7952c87c47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2283009b178044279468c7952c87c47e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45359 } } }
I20260620 19:00:26.377491 5890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:26.377610 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:26.378247 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:00:26.380861 5890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:26.386276 5890 catalog_manager.cc:1347] Generated new cluster ID: 69b38bd29664456f8572682461e54f45
I20260620 19:00:26.386360 5890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:26.398451 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:26.392882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.2.208.190:45359)
0620 19:00:26.393014 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:26.393019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:26.393280 (+ 261us) server_negotiation.cc:407] Connection header received
0620 19:00:26.393401 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:26.393404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:26.393471 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:26.393564 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:26.394088 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:26.394894 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:26.396021 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:26.396393 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:26.396498 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:26.397377 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:26.397381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:26.397384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:26.397997 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:26.398034 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:26.398053 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:26.398139 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:26.398238 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:00:26.398292 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:00:26.404428 5890 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:26.405300 5890 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:26.419160 5890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2283009b178044279468c7952c87c47e: Generated new TSK 0
I20260620 19:00:26.420010 5890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:00:26.432264 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5808
2026-06-20T19:00:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1414 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5909](info): set up 2 sockets
Jun 20 19:00:26 dist-test-slave-tsm4 krb5kdc[5909](info): commencing operation
krb5kdc: starting...
W20260620 19:00:27.463290 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:27 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:27.580440 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36901
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:36901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:27.687481 5922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:27.687731 5922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:27.687788 5922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:27.691666 5922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:27.691712 5922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:27.691730 5922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:27.691747 5922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:27.691762 5922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:27.696756 5922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:36901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:27.697713 5922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:27.698649 5922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:27.706960 5927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:27.706949 5930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:27.706959 5928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:27.707226 5922 server_base.cc:1034] running on GCE node
I20260620 19:00:27.707842 5922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:27.708922 5922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:27.710098 5922 hybrid_clock.cc:648] HybridClock initialized: now 1781982027710086 us; error 31 us; skew 500 ppm
Jun 20 19:00:27 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982027, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:27.712741 5922 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:27.713613 5922 webserver.cc:460] Webserver started at http://127.2.208.190:36711/ using document root <none> and password file <none>
I20260620 19:00:27.714205 5922 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:27.714288 5922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:27.714506 5922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:27.716537 5922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "67b9d6301cdd44efac0119e1836283b0"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "9db985bd854a0931c7e6e47b6104900f"
server_key_iv: "6302543e683ff9f454b8fa06344e9473"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.717090 5922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "67b9d6301cdd44efac0119e1836283b0"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "9db985bd854a0931c7e6e47b6104900f"
server_key_iv: "6302543e683ff9f454b8fa06344e9473"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.720607 5922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:27.722898 5937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:27.724140 5922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:27.724291 5922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "67b9d6301cdd44efac0119e1836283b0"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "9db985bd854a0931c7e6e47b6104900f"
server_key_iv: "6302543e683ff9f454b8fa06344e9473"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.724400 5922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:27.738540 5922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:27.741357 5922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:27.741554 5922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:27.748555 5922 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:36901
I20260620 19:00:27.748814 5989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:36901 every 8 connection(s)
I20260620 19:00:27.749543 5922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:27.752316 5990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:27.756217 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 5922
I20260620 19:00:27.756346 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:27.756624 2882 external_mini_cluster.cc:1428] Setting key b793af97af60231bedccce514b2eba25
I20260620 19:00:27.757990 5990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0: Bootstrap starting.
I20260620 19:00:27.760720 5990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:27.761610 5990 log.cc:824] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:27.763602 5990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0: No bootstrap required, opened a new log
Jun 20 19:00:27 dist-test-slave-tsm4 krb5kdc[5909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:27.767947 5990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } }
I20260620 19:00:27.768162 5990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:27.768229 5990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b9d6301cdd44efac0119e1836283b0, State: Initialized, Role: FOLLOWER
I20260620 19:00:27.768694 5990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } }
I20260620 19:00:27.768812 5990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:27.768890 5990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:27.768993 5990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:27.769857 5990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } }
I20260620 19:00:27.770251 5990 leader_election.cc:302] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b9d6301cdd44efac0119e1836283b0; no voters:
I20260620 19:00:27.770646 5990 leader_election.cc:288] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:27.770716 5995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:27.770889 5995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [term 1 LEADER]: Becoming Leader. State: Replica: 67b9d6301cdd44efac0119e1836283b0, State: Running, Role: LEADER
I20260620 19:00:27.771302 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:27.757924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.2.208.190:36901)
0620 19:00:27.758360 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:27.758377 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:27.758423 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:00:27.759387 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:27.759411 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:27.759818 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:27.760180 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:27.760819 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:27.762192 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:27.763038 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:27.763727 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:27.763894 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:27.766474 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:27.766478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:27.766491 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:27.766517 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:27.768234 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:27.768682 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:27.768688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:27.768693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:27.768838 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:27.769149 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:27.769155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:27.769157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:27.769549 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:27.769738 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:27.770037 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 19:00:27.770324 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":154,"threads_started":1}
I20260620 19:00:27.771323 5995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } }
I20260620 19:00:27.771920 5990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:27.773115 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b9d6301cdd44efac0119e1836283b0. Latest consensus state: current_term: 1 leader_uuid: "67b9d6301cdd44efac0119e1836283b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } } }
I20260620 19:00:27.773141 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b9d6301cdd44efac0119e1836283b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9d6301cdd44efac0119e1836283b0" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36901 } } }
I20260620 19:00:27.773229 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:27.773233 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:27.773633 6004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:27.776821 6004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:27.782725 6004 catalog_manager.cc:1347] Generated new cluster ID: 8ee50b85fd8f4156995796d907cdb1df
I20260620 19:00:27.782801 6004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:27.794126 6004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:27.795328 6004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:27.802462 6004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b9d6301cdd44efac0119e1836283b0: Generated new TSK 0
I20260620 19:00:27.803331 6004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:27.835582 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:27.945308 6016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:27.945554 6016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:27.945595 6016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:27.949483 6016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:27.949536 6016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:27.949592 6016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:27.954582 6016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:27.955623 6016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:27.956540 6016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:27.965585 6022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:27.965606 6021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:27.965591 6024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:27.965806 6016 server_base.cc:1034] running on GCE node
I20260620 19:00:27.966205 6016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:27.966773 6016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:27.967931 6016 hybrid_clock.cc:648] HybridClock initialized: now 1781982027967892 us; error 45 us; skew 500 ppm
Jun 20 19:00:27 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982027, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:27.970475 6016 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:27.971261 6016 webserver.cc:460] Webserver started at http://127.2.208.129:42731/ using document root <none> and password file <none>
I20260620 19:00:27.971825 6016 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:27.971910 6016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:27.972129 6016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:27.974251 6016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "8ebd35f2b1c842bc9724ecd9567044da"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "1d3563e479040e805fb41497306d8184"
server_key_iv: "f0dada2e51f7f037a3b46c7b556c1b76"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.974785 6016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "8ebd35f2b1c842bc9724ecd9567044da"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "1d3563e479040e805fb41497306d8184"
server_key_iv: "f0dada2e51f7f037a3b46c7b556c1b76"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.978330 6016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:27.980621 6031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:27.981745 6016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:27.981868 6016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "8ebd35f2b1c842bc9724ecd9567044da"
format_stamp: "Formatted at 2026-06-20 19:00:27 on dist-test-slave-tsm4"
server_key: "1d3563e479040e805fb41497306d8184"
server_key_iv: "f0dada2e51f7f037a3b46c7b556c1b76"
server_key_version: "encryptionkey@0"
I20260620 19:00:27.981966 6016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:28.007807 6016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:28.010597 6016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:28.010793 6016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:28.011423 6016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:28.012219 6016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:28.012264 6016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.012343 6016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:28.012373 6016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.021195 6016 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:42435
I20260620 19:00:28.021217 6144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:42435 every 8 connection(s)
I20260620 19:00:28.022056 6016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982027, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:28.031702 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6016
I20260620 19:00:28.031832 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:28.032176 2882 external_mini_cluster.cc:1428] Setting key 371f49ce532e24aa759e3ebd1a47abae
I20260620 19:00:28.035507 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.023815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55415 (local address 127.2.208.190:36901)
0620 19:00:28.023951 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:28.023955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:28.024720 (+ 765us) server_negotiation.cc:407] Connection header received
0620 19:00:28.026123 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:28.026126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:28.026170 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:28.026243 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:28.027480 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.028040 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.029185 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.029415 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.029499 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.031820 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:28.031823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:28.031826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:28.031853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:28.033435 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.033918 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.033923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.033925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.034012 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.034310 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.034316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.034321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.034571 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:28.034665 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:28.035157 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:00:28.035293 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:00:28.036338 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.024172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36901 (local address 127.2.208.129:55415)
0620 19:00:28.024578 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:28.024618 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:28.025935 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:28.026397 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:28.026404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:28.026740 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:28.027298 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.027312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.028149 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.028151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.029086 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.029099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.029541 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.029544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.029682 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.030242 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:28.030258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:28.031659 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:28.033560 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.033564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.033576 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.033820 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.034100 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.034102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.034103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.034227 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.034670 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:28.034675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:28.034901 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 19:00:28.035121 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 19:00:28.035479 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260620 19:00:28.037534 6145 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36901
I20260620 19:00:28.037750 6145 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:28.038185 6145 heartbeater.cc:511] Master 127.2.208.190:36901 requested a full tablet report, sending...
I20260620 19:00:28.039624 5954 ts_manager.cc:194] Registered new tserver with Master: 8ebd35f2b1c842bc9724ecd9567044da (127.2.208.129:42435)
I20260620 19:00:28.040700 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:55415
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:28.056392 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:28.164863 6150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:28.165096 6150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:28.165189 6150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:28.169034 6150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:28.169087 6150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:28.169181 6150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:28.174507 6150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:28.175540 6150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:28.176556 6150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:28.186190 6155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:28.186337 6150 server_base.cc:1034] running on GCE node
W20260620 19:00:28.186223 6156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:28.186182 6158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:28.186841 6150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:28.187508 6150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:28.188683 6150 hybrid_clock.cc:648] HybridClock initialized: now 1781982028188661 us; error 38 us; skew 500 ppm
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:28.191447 6150 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:28.192272 6150 webserver.cc:460] Webserver started at http://127.2.208.130:36231/ using document root <none> and password file <none>
I20260620 19:00:28.192849 6150 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:28.192931 6150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:28.193148 6150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:28.195165 6150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "0652eedf39b346749d01a16ad22e743e"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "fa57a205b86f927a8fb859b6357f1b24"
server_key_iv: "96cdf62384ca4287186261aa39fd1f45"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.195760 6150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "0652eedf39b346749d01a16ad22e743e"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "fa57a205b86f927a8fb859b6357f1b24"
server_key_iv: "96cdf62384ca4287186261aa39fd1f45"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.199503 6150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:28.201978 6165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.203156 6150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:28.203279 6150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "0652eedf39b346749d01a16ad22e743e"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "fa57a205b86f927a8fb859b6357f1b24"
server_key_iv: "96cdf62384ca4287186261aa39fd1f45"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.203418 6150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:28.232852 6150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:28.236243 6150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:28.236449 6150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:28.237043 6150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:28.237869 6150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:28.237939 6150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.238003 6150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:28.238049 6150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.247148 6150 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:41579
I20260620 19:00:28.247165 6278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:41579 every 8 connection(s)
I20260620 19:00:28.248157 6150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:28.252164 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6150
I20260620 19:00:28.252312 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:28.252624 2882 external_mini_cluster.cc:1428] Setting key d07d882f9245b850a592739c1f55310e
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:00:28.263262 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.250025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:51193 (local address 127.2.208.190:36901)
0620 19:00:28.250195 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:28.250198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:28.250974 (+ 776us) server_negotiation.cc:407] Connection header received
0620 19:00:28.252534 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:28.252538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:28.252596 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:28.252699 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:28.254289 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.255067 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.256266 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.256503 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.256613 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.259748 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:28.259751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:28.259753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:28.259776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:28.261163 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.261750 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.261752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.261754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.261821 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.262206 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.262208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.262209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.262364 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:28.262471 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:28.262964 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 19:00:28.263085 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 19:00:28.264117 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.250414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36901 (local address 127.2.208.130:51193)
0620 19:00:28.250829 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:28.250870 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:28.252269 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:28.252885 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:28.252892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:28.253363 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:28.254033 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.254054 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.255181 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.255183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.256144 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.256152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.256651 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.256655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.256821 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.257658 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:28.257675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:28.259575 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:28.261278 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.261284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.261300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.261653 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.261916 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.261921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.261924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.262110 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.262469 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:28.262476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:28.262710 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:00:28.262913 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 19:00:28.263276 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
I20260620 19:00:28.265321 6279 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36901
I20260620 19:00:28.265564 6279 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:28.266059 6279 heartbeater.cc:511] Master 127.2.208.190:36901 requested a full tablet report, sending...
I20260620 19:00:28.267179 5954 ts_manager.cc:194] Registered new tserver with Master: 0652eedf39b346749d01a16ad22e743e (127.2.208.130:41579)
I20260620 19:00:28.267935 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:51193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:28.280330 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:28.387537 6284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:28.387828 6284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:28.387897 6284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:28.391937 6284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:28.392019 6284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:28.392102 6284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:28.397393 6284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:28.398442 6284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:28.399513 6284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:28.408739 6290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:28.408782 6292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:28.408785 6289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:28.408888 6284 server_base.cc:1034] running on GCE node
I20260620 19:00:28.409404 6284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:28.409977 6284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:28.411127 6284 hybrid_clock.cc:648] HybridClock initialized: now 1781982028411096 us; error 44 us; skew 500 ppm
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:28.413684 6284 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:28.414525 6284 webserver.cc:460] Webserver started at http://127.2.208.131:34057/ using document root <none> and password file <none>
I20260620 19:00:28.415063 6284 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:28.415119 6284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:28.415344 6284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:28.417331 6284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "dda6cb2a0e10401a94ab471b2d566125"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "a55f4b785c21c4739192303aea959b85"
server_key_iv: "df971dad2857b419d285c7b61c1648f0"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.417811 6284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "dda6cb2a0e10401a94ab471b2d566125"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "a55f4b785c21c4739192303aea959b85"
server_key_iv: "df971dad2857b419d285c7b61c1648f0"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.421291 6284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:28.423494 6299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.424643 6284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:28.424764 6284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "dda6cb2a0e10401a94ab471b2d566125"
format_stamp: "Formatted at 2026-06-20 19:00:28 on dist-test-slave-tsm4"
server_key: "a55f4b785c21c4739192303aea959b85"
server_key_iv: "df971dad2857b419d285c7b61c1648f0"
server_key_version: "encryptionkey@0"
I20260620 19:00:28.424865 6284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:28.436277 6284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:28.438848 6284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:28.439038 6284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:28.439654 6284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:28.440415 6284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:28.440486 6284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.440552 6284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:28.440584 6284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:28.448750 6284 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:40633
I20260620 19:00:28.448769 6412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:40633 every 8 connection(s)
I20260620 19:00:28.449664 6284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:28.456172 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6284
I20260620 19:00:28.456298 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:28.456571 2882 external_mini_cluster.cc:1428] Setting key 8f756152760bee59bbb81a10c0bfb1af
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:28.462653 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.451353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:38289 (local address 127.2.208.190:36901)
0620 19:00:28.451535 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:28.451541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:28.452276 (+ 735us) server_negotiation.cc:407] Connection header received
0620 19:00:28.453664 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:28.453667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:28.453721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:28.453831 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:28.454930 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.455509 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.456672 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.456934 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.457042 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.459489 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:28.459493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:28.459496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:28.459523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:28.460822 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.461312 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.461317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.461318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.461390 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.461695 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.461700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.461701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.461864 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:28.461968 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:28.462398 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 19:00:28.462506 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:00:28.463481 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.451738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36901 (local address 127.2.208.131:38289)
0620 19:00:28.452144 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:28.452183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:28.453463 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:28.453995 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:28.454002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:28.454332 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:28.454784 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.454796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.455632 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.455635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.456552 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:28.456561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.457090 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.457094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:28.457288 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.457931 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:28.457950 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:28.459328 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:28.460961 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.460966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.460980 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.461218 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.461480 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:28.461483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:28.461484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:28.461611 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:28.461966 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:28.461972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:28.462193 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 19:00:28.462361 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 19:00:28.462654 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260620 19:00:28.464443 6413 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36901
I20260620 19:00:28.464648 6413 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:28.465063 6413 heartbeater.cc:511] Master 127.2.208.190:36901 requested a full tablet report, sending...
I20260620 19:00:28.465931 5954 ts_manager.cc:194] Registered new tserver with Master: dda6cb2a0e10401a94ab471b2d566125 (127.2.208.131:40633)
I20260620 19:00:28.466439 5954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:38289
I20260620 19:00:28.470016 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[5909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982028, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:28.494539 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:28.486493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.2.208.190:36901)
0620 19:00:28.486649 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:28.486655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:28.486791 (+ 136us) server_negotiation.cc:407] Connection header received
0620 19:00:28.486960 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:28.486963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:28.487003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:28.487083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:28.487825 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.488342 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.489178 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:28.489397 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:28.489502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:28.491778 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:28.491782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:28.491784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:28.491806 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:28.492946 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.493338 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.493343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.493345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.493419 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:28.493672 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:28.493678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:28.493681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:28.493897 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:28.494041 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:28.494266 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:00:28.494378 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 19:00:28.494920 5954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45356
I20260620 19:00:28.495812 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6016
I20260620 19:00:28.502278 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6150
I20260620 19:00:28.507550 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6284
I20260620 19:00:28.512334 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 5922
2026-06-20T19:00:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2081 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[6425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[6425](info): set up 2 sockets
Jun 20 19:00:28 dist-test-slave-tsm4 krb5kdc[6425](info): commencing operation
krb5kdc: starting...
W20260620 19:00:29.543470 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:29 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:29.664062 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44263
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:29.770510 6438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:29.770764 6438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:29.770803 6438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:29.774756 6438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:29.774812 6438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:29.774832 6438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:29.774848 6438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:29.774890 6438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:29.779975 6438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:29.780964 6438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:29.781955 6438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:29.790220 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:29.790220 6438 server_base.cc:1034] running on GCE node
W20260620 19:00:29.790205 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:29.790205 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:29.791049 6438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:29.792217 6438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:29.793396 6438 hybrid_clock.cc:648] HybridClock initialized: now 1781982029793376 us; error 48 us; skew 500 ppm
Jun 20 19:00:29 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982029, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:29.795991 6438 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:29.796782 6438 webserver.cc:460] Webserver started at http://127.2.208.190:36599/ using document root <none> and password file <none>
I20260620 19:00:29.797314 6438 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:29.797401 6438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:29.797622 6438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:29.799690 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "66c18f5fdb5a4774b2650bd280bfd214"
format_stamp: "Formatted at 2026-06-20 19:00:29 on dist-test-slave-tsm4"
server_key: "ad78664620aa96573b0cdb67de67f4da"
server_key_iv: "e9bcd019b9e92759d201695c54d45e23"
server_key_version: "encryptionkey@0"
I20260620 19:00:29.800226 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "66c18f5fdb5a4774b2650bd280bfd214"
format_stamp: "Formatted at 2026-06-20 19:00:29 on dist-test-slave-tsm4"
server_key: "ad78664620aa96573b0cdb67de67f4da"
server_key_iv: "e9bcd019b9e92759d201695c54d45e23"
server_key_version: "encryptionkey@0"
I20260620 19:00:29.803824 6438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:29.806147 6453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:29.807273 6438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:29.807441 6438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "66c18f5fdb5a4774b2650bd280bfd214"
format_stamp: "Formatted at 2026-06-20 19:00:29 on dist-test-slave-tsm4"
server_key: "ad78664620aa96573b0cdb67de67f4da"
server_key_iv: "e9bcd019b9e92759d201695c54d45e23"
server_key_version: "encryptionkey@0"
I20260620 19:00:29.807552 6438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:29.826393 6438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:29.829278 6438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:29.829471 6438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:29.836189 6438 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44263
I20260620 19:00:29.836211 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44263 every 8 connection(s)
I20260620 19:00:29.837069 6438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:29.839603 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6438
I20260620 19:00:29.839634 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:29.839694 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:29.839973 2882 external_mini_cluster.cc:1428] Setting key 87524c6c0a80bc7d1126f14df44ddef0
I20260620 19:00:29.846501 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: Bootstrap starting.
I20260620 19:00:29.848809 6506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: Neither blocks nor log segments found. Creating new log.
Jun 20 19:00:29 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:29.849586 6506 log.cc:824] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:29.851303 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: No bootstrap required, opened a new log
I20260620 19:00:29.854036 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:29.841329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.2.208.190:44263)
0620 19:00:29.841740 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:29.841757 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:29.841796 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:00:29.842932 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:29.842954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:29.843348 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:29.843717 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:29.844360 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:29.845845 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:29.846666 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:29.847126 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:29.847294 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:29.849761 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:29.849766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:29.849778 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:29.849803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:29.851483 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:29.851888 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:29.851894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:29.851909 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:29.852011 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:29.852274 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:29.852278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:29.852280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:29.852656 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:29.852856 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:29.853106 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:00:29.853369 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260620 19:00:29.854768 6506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } }
I20260620 19:00:29.855020 6506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:29.855078 6506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c18f5fdb5a4774b2650bd280bfd214, State: Initialized, Role: FOLLOWER
I20260620 19:00:29.855540 6506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } }
I20260620 19:00:29.855685 6506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:29.855746 6506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:29.855834 6506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:29.856719 6506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } }
I20260620 19:00:29.856989 6506 leader_election.cc:302] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c18f5fdb5a4774b2650bd280bfd214; no voters:
I20260620 19:00:29.857213 6506 leader_election.cc:288] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:29.857326 6511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:29.857559 6511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [term 1 LEADER]: Becoming Leader. State: Replica: 66c18f5fdb5a4774b2650bd280bfd214, State: Running, Role: LEADER
I20260620 19:00:29.857897 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } }
I20260620 19:00:29.858446 6506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:29.859503 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c18f5fdb5a4774b2650bd280bfd214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } } }
I20260620 19:00:29.859619 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:29.859894 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c18f5fdb5a4774b2650bd280bfd214. Latest consensus state: current_term: 1 leader_uuid: "66c18f5fdb5a4774b2650bd280bfd214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c18f5fdb5a4774b2650bd280bfd214" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44263 } } }
I20260620 19:00:29.860031 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214 [sys.catalog]: This master's current role is: LEADER
W20260620 19:00:29.862994 6526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:29.863091 6526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:29.863157 6520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:29.863886 6520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:29.869029 6520 catalog_manager.cc:1347] Generated new cluster ID: 2096fbcacf3b42e6aa5165d43cc1cd93
I20260620 19:00:29.869096 6520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:29.882746 6520 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:29.883852 6520 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:29.890121 6520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c18f5fdb5a4774b2650bd280bfd214: Generated new TSK 0
I20260620 19:00:29.890764 6520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:29.921634 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:30.029829 6532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:30.030110 6532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:30.030181 6532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:30.034205 6532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:30.034287 6532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:30.034373 6532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:30.039354 6532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:30.040375 6532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:30.041330 6532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:30.050587 6540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:30.050597 6538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:30.050704 6532 server_base.cc:1034] running on GCE node
W20260620 19:00:30.050597 6537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:30.051218 6532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:30.051875 6532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:30.053061 6532 hybrid_clock.cc:648] HybridClock initialized: now 1781982030053052 us; error 32 us; skew 500 ppm
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:30.055665 6532 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:30.056481 6532 webserver.cc:460] Webserver started at http://127.2.208.129:35113/ using document root <none> and password file <none>
I20260620 19:00:30.057073 6532 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:30.057159 6532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:30.057379 6532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:30.059612 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "8a2de822bcc540d6ae0d7d58e1c37fb9"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "10b186ab0143bad7041dec8b1a6a3353"
server_key_iv: "a681d44ba67dd4c445a2fca8f8974efd"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.060207 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "8a2de822bcc540d6ae0d7d58e1c37fb9"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "10b186ab0143bad7041dec8b1a6a3353"
server_key_iv: "a681d44ba67dd4c445a2fca8f8974efd"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.063802 6532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:30.066123 6547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.067313 6532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:30.067458 6532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "8a2de822bcc540d6ae0d7d58e1c37fb9"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "10b186ab0143bad7041dec8b1a6a3353"
server_key_iv: "a681d44ba67dd4c445a2fca8f8974efd"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.067561 6532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:30.085285 6532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:30.088083 6532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:30.088275 6532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:30.088845 6532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:30.089641 6532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:30.089711 6532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.089784 6532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:30.089825 6532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.098030 6532 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:33021
I20260620 19:00:30.098055 6660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:33021 every 8 connection(s)
I20260620 19:00:30.098845 6532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:30.107905 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6532
I20260620 19:00:30.108023 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:30.108319 2882 external_mini_cluster.cc:1428] Setting key 3a9bac812b6990fd2e37c6a130401979
I20260620 19:00:30.112965 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.100554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:41133 (local address 127.2.208.190:44263)
0620 19:00:30.100686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:30.100689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:30.101489 (+ 800us) server_negotiation.cc:407] Connection header received
0620 19:00:30.103004 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:30.103008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:30.103064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:30.103167 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:30.104336 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.104859 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.105968 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.106207 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.106312 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.108956 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:30.108959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:30.108961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:30.108983 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:30.110540 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.111139 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.111142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.111143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.111213 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.111674 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.111679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.111682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.111947 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:30.112049 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:30.112581 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 19:00:30.112713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:00:30.113595 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.100933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44263 (local address 127.2.208.129:41133)
0620 19:00:30.101343 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:30.101383 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:30.102798 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:30.103329 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:30.103336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:30.103738 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:30.104199 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.104211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.104985 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.104993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.105844 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.105852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.106349 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.106353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.106512 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.107087 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:30.107102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:30.108777 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:30.110659 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.110668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.110685 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.111052 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.111304 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.111309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.111313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.111554 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.112064 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:30.112070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:30.112295 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 19:00:30.112482 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 19:00:30.112722 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260620 19:00:30.114777 6661 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44263
I20260620 19:00:30.115038 6661 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:30.115564 6661 heartbeater.cc:511] Master 127.2.208.190:44263 requested a full tablet report, sending...
I20260620 19:00:30.117137 6470 ts_manager.cc:194] Registered new tserver with Master: 8a2de822bcc540d6ae0d7d58e1c37fb9 (127.2.208.129:33021)
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:30.118635 6470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:41133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:30.133808 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:30.240787 6666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:30.241025 6666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:30.241065 6666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:30.245005 6666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:30.245074 6666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:30.245128 6666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:30.250492 6666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:30.251530 6666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:30.252478 6666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:30.262387 6674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:30.262394 6672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:30.262403 6671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:30.262537 6666 server_base.cc:1034] running on GCE node
I20260620 19:00:30.262996 6666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:30.263624 6666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:30.264782 6666 hybrid_clock.cc:648] HybridClock initialized: now 1781982030264764 us; error 31 us; skew 500 ppm
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:30.267590 6666 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:30.268358 6666 webserver.cc:460] Webserver started at http://127.2.208.130:41037/ using document root <none> and password file <none>
I20260620 19:00:30.268858 6666 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:30.268916 6666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:30.269132 6666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:30.271183 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "8607d674460641bb8973344ddacbb547"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "13dfaa6157b9fabb8b118b30f4246f5f"
server_key_iv: "86085f02f01861c07dbcb39122bf344f"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.271724 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "8607d674460641bb8973344ddacbb547"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "13dfaa6157b9fabb8b118b30f4246f5f"
server_key_iv: "86085f02f01861c07dbcb39122bf344f"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.276902 6666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 19:00:30.280260 6681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.282222 6666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 19:00:30.282310 6666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "8607d674460641bb8973344ddacbb547"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "13dfaa6157b9fabb8b118b30f4246f5f"
server_key_iv: "86085f02f01861c07dbcb39122bf344f"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.282446 6666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:30.304409 6666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:30.308400 6666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:30.308653 6666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:30.309480 6666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:30.310684 6666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:30.310755 6666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.310822 6666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:30.310863 6666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.322194 6666 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:34895
I20260620 19:00:30.322252 6794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:34895 every 8 connection(s)
I20260620 19:00:30.323091 6666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:30.330015 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6666
I20260620 19:00:30.330137 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:30.330412 2882 external_mini_cluster.cc:1428] Setting key 39f5804b7d93d091a13ba11ade0e4575
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:30.336799 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.324835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:47591 (local address 127.2.208.190:44263)
0620 19:00:30.324968 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:30.324971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:30.325710 (+ 739us) server_negotiation.cc:407] Connection header received
0620 19:00:30.327049 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:30.327052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:30.327108 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:30.327208 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:30.328350 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.328917 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.330030 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.330314 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.330445 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.332845 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:30.332849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:30.332851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:30.332876 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:30.334579 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.335069 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.335075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.335080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.335170 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.335549 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.335555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.335560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.335812 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:30.335932 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:30.336468 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 19:00:30.336597 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:30.337522 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.325169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44263 (local address 127.2.208.130:47591)
0620 19:00:30.325575 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:30.325613 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:30.326847 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:30.327406 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:30.327415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:30.327748 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:30.328195 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.328206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.329034 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.329038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.329910 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.329918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.330479 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.330483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.330645 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.331170 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:30.331185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:30.332690 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:30.334697 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.334702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.334713 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.334974 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.335263 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.335265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.335267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.335455 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.335936 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:30.335941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:30.336202 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 19:00:30.336368 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 19:00:30.336606 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
I20260620 19:00:30.338642 6795 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44263
I20260620 19:00:30.338907 6795 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:30.339522 6795 heartbeater.cc:511] Master 127.2.208.190:44263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:00:30.340648 6470 ts_manager.cc:194] Registered new tserver with Master: 8607d674460641bb8973344ddacbb547 (127.2.208.130:34895)
I20260620 19:00:30.341221 6470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:47591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:30.356420 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:30.469312 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:30.469580 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:30.469648 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:30.473980 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:30.474043 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:30.474134 6800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:30.479852 6800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:30.480906 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:30.481948 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:30.492262 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:30.492292 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:30.492254 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:30.492529 6800 server_base.cc:1034] running on GCE node
I20260620 19:00:30.492959 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:30.493578 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:30.494760 6800 hybrid_clock.cc:648] HybridClock initialized: now 1781982030494711 us; error 65 us; skew 500 ppm
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:30.497545 6800 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:30.498386 6800 webserver.cc:460] Webserver started at http://127.2.208.131:43017/ using document root <none> and password file <none>
I20260620 19:00:30.498945 6800 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:30.499032 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:30.499257 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:30.501390 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "e5e0cbaf537d4b13bfdbcf5cef9b7f31"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "b081f897bedc3aa1b26444e26de47a24"
server_key_iv: "f8fc0d0341e9fa9dfc12975801af606b"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.501932 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "e5e0cbaf537d4b13bfdbcf5cef9b7f31"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "b081f897bedc3aa1b26444e26de47a24"
server_key_iv: "f8fc0d0341e9fa9dfc12975801af606b"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.505611 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:30.508929 6815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.510208 6800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:00:30.510335 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e5e0cbaf537d4b13bfdbcf5cef9b7f31"
format_stamp: "Formatted at 2026-06-20 19:00:30 on dist-test-slave-tsm4"
server_key: "b081f897bedc3aa1b26444e26de47a24"
server_key_iv: "f8fc0d0341e9fa9dfc12975801af606b"
server_key_version: "encryptionkey@0"
I20260620 19:00:30.510432 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:30.530236 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:30.533190 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:30.533344 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:30.533878 6800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:30.534665 6800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:30.534710 6800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.534749 6800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:30.534770 6800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:30.543473 6800 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:41379
I20260620 19:00:30.543572 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:41379 every 8 connection(s)
I20260620 19:00:30.544333 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:30.551874 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6800
I20260620 19:00:30.552021 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:30.552291 2882 external_mini_cluster.cc:1428] Setting key 9aabd2bd94f6108b984e6ec847ce500e
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:30.558007 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.546475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:51529 (local address 127.2.208.190:44263)
0620 19:00:30.546639 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:30.546643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:30.547441 (+ 798us) server_negotiation.cc:407] Connection header received
0620 19:00:30.548805 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:30.548808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:30.548862 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:30.548950 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:30.550076 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.550610 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.551796 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.552070 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.552183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.554867 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:30.554870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:30.554872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:30.554895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:30.556146 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.556729 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.556732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.556733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.556800 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.557156 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.557158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.557159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.557312 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:30.557383 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:30.557804 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 19:00:30.557896 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:00:30.558899 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.546855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44263 (local address 127.2.208.131:51529)
0620 19:00:30.547275 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:30.547317 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:30.548605 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:30.549128 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:30.549135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:30.549480 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:30.549939 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.549952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.550731 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.550734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.551674 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:30.551682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.552216 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.552219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:30.552411 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.553054 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:30.553075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:30.554735 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:30.556242 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.556247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.556258 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.556633 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.556878 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:30.556883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:30.556885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:30.557053 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:30.557398 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:30.557402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:30.557615 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 19:00:30.557763 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 19:00:30.558065 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260620 19:00:30.560043 6929 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44263
I20260620 19:00:30.560235 6929 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:30.560729 6929 heartbeater.cc:511] Master 127.2.208.190:44263 requested a full tablet report, sending...
I20260620 19:00:30.561698 6470 ts_manager.cc:194] Registered new tserver with Master: e5e0cbaf537d4b13bfdbcf5cef9b7f31 (127.2.208.131:41379)
I20260620 19:00:30.562247 6470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:51529
I20260620 19:00:30.565712 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:30.653991 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.644602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.2.208.129:33021)
0620 19:00:30.645086 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:30.645096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:30.645126 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:00:30.645191 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:30.645200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:30.645460 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:30.645628 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:30.646300 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.647104 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.647982 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.648475 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.648561 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.650790 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:30.650821 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:30.650843 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:30.650867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:30.652401 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.652772 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.652777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.652778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.652851 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.653076 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.653078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.653079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.653308 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:30.653470 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:30.653642 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:00:30.653797 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":129,"threads_started":1}
Jun 20 19:00:30 dist-test-slave-tsm4 krb5kdc[6425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:30.664820 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:30.656823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.2.208.190:44263)
0620 19:00:30.656963 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:30.656966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:30.656994 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:00:30.657117 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:30.657119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:30.657175 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:30.657259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:30.657789 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.658336 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.659227 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:30.659492 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:30.659571 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:30.661762 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:30.661765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:30.661768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:30.661789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:30.663157 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.663622 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.663626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.663628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.663705 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:30.663978 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:30.663983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:30.663985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:30.664190 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:30.664330 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:30.664545 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:00:30.664662 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 19:00:30.665218 6470 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57024
I20260620 19:00:30.665805 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6532
I20260620 19:00:30.671187 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6666
I20260620 19:00:30.676359 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6800
I20260620 19:00:30.681277 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6438
2026-06-20T19:00:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2169 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:32 dist-test-slave-tsm4 krb5kdc[6941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:32 dist-test-slave-tsm4 krb5kdc[6941](info): set up 2 sockets
Jun 20 19:00:32 dist-test-slave-tsm4 krb5kdc[6941](info): commencing operation
krb5kdc: starting...
W20260620 19:00:33.990907 2882 mini_kdc.cc:121] Time spent starting KDC: real 3.287s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 19:00:34.201448 2882 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.210s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 19:00:34.510633 2882 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.309s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:34 dist-test-slave-tsm4 krb5kdc[6941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:34.587990 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43073
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:34.699550 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:34.699820 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:34.699895 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:34.704013 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:34.704074 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:34.704095 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:34.704113 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:34.704160 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:34.709410 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:34.710597 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:34.711663 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:34.719990 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:34.720055 6954 server_base.cc:1034] running on GCE node
W20260620 19:00:34.720005 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:34.720000 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:34.720744 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:34.721782 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:34.722936 6954 hybrid_clock.cc:648] HybridClock initialized: now 1781982034722915 us; error 35 us; skew 500 ppm
Jun 20 19:00:34 dist-test-slave-tsm4 krb5kdc[6941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982034, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:34.725648 6954 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:34.726399 6954 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:34.726464 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:34.726670 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:34.728745 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "23d678f9412e42d89ce8fb398e85d40a"
format_stamp: "Formatted at 2026-06-20 19:00:34 on dist-test-slave-tsm4"
server_key: "9084220987af2778b056f7a581375cd0"
server_key_iv: "ba4f1527718e64621a3b7340c208bb07"
server_key_version: "encryptionkey@0"
I20260620 19:00:34.729307 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "23d678f9412e42d89ce8fb398e85d40a"
format_stamp: "Formatted at 2026-06-20 19:00:34 on dist-test-slave-tsm4"
server_key: "9084220987af2778b056f7a581375cd0"
server_key_iv: "ba4f1527718e64621a3b7340c208bb07"
server_key_version: "encryptionkey@0"
I20260620 19:00:34.732897 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:34.735198 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:34.736542 6954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:34.736680 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "23d678f9412e42d89ce8fb398e85d40a"
format_stamp: "Formatted at 2026-06-20 19:00:34 on dist-test-slave-tsm4"
server_key: "9084220987af2778b056f7a581375cd0"
server_key_iv: "ba4f1527718e64621a3b7340c208bb07"
server_key_version: "encryptionkey@0"
I20260620 19:00:34.736768 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:34.769172 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:34.772176 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:34.772385 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:34.779222 6954 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:43073
I20260620 19:00:34.779507 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:43073 every 8 connection(s)
I20260620 19:00:34.779932 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:34.782626 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:34.783748 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 6954
I20260620 19:00:34.783886 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:34.784143 2882 external_mini_cluster.cc:1428] Setting key baae0823ad850d529a7cdd8fab1d76fa
I20260620 19:00:34.789251 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: Bootstrap starting.
I20260620 19:00:34.791754 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:34.792472 7021 log.cc:824] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:00:34 dist-test-slave-tsm4 krb5kdc[6941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:34.794694 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: No bootstrap required, opened a new log
I20260620 19:00:34.799008 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:34.785558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.2.208.190:43073)
0620 19:00:34.785960 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:34.785982 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:34.786023 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:00:34.786822 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:34.786844 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:34.787268 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:34.787652 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:34.788281 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:34.789686 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:34.790509 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:34.791140 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:34.791304 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:34.793931 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:34.793937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:34.793951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:34.793980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:34.795948 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:34.796340 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:34.796345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:34.796351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:34.796489 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:34.796734 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:34.796737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:34.796739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:34.797362 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:34.797547 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:34.797732 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:00:34.798016 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20260620 19:00:34.799351 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } }
I20260620 19:00:34.799636 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:34.799692 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d678f9412e42d89ce8fb398e85d40a, State: Initialized, Role: FOLLOWER
I20260620 19:00:34.800140 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } }
I20260620 19:00:34.800303 7021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:34.800366 7021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:34.800460 7021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:34.801680 7021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } }
I20260620 19:00:34.801973 7021 leader_election.cc:302] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d678f9412e42d89ce8fb398e85d40a; no voters:
I20260620 19:00:34.802204 7021 leader_election.cc:288] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:34.802315 7026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:34.802543 7026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [term 1 LEADER]: Becoming Leader. State: Replica: 23d678f9412e42d89ce8fb398e85d40a, State: Running, Role: LEADER
I20260620 19:00:34.802850 7026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } }
I20260620 19:00:34.803406 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:34.804456 7028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d678f9412e42d89ce8fb398e85d40a. Latest consensus state: current_term: 1 leader_uuid: "23d678f9412e42d89ce8fb398e85d40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } } }
I20260620 19:00:34.804584 7028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:34.804965 7035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:34.805538 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d678f9412e42d89ce8fb398e85d40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d678f9412e42d89ce8fb398e85d40a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43073 } } }
I20260620 19:00:34.805642 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:34.807973 7035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:34.813638 7035 catalog_manager.cc:1347] Generated new cluster ID: 6e2aa984dce54983aed8bf8287fe6558
I20260620 19:00:34.813702 7035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:34.843214 7035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:34.844251 7035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:34.850813 7035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: Generated new TSK 0
I20260620 19:00:34.851528 7035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:34.880383 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:34.991007 7047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:34.991320 7047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:34.991418 7047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:34.995431 7047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:34.995512 7047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:34.995600 7047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:35.000959 7047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:35.002081 7047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:35.003131 7047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:35.012809 7053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.012820 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.012856 7047 server_base.cc:1034] running on GCE node
W20260620 19:00:35.012830 7052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.013531 7047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:35.014178 7047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:35.015395 7047 hybrid_clock.cc:648] HybridClock initialized: now 1781982035015348 us; error 26 us; skew 500 ppm
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:35.018046 7047 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:35.018694 7047 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:35.018751 7047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:35.018919 7047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:35.020881 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "2834c08228574d1db3d9135a437af4ed"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "5b575e49ee91d0f3520bffb53281df6c"
server_key_iv: "9fcb67d11670fbb4cc0133cefb7aa4db"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.021370 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "2834c08228574d1db3d9135a437af4ed"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "5b575e49ee91d0f3520bffb53281df6c"
server_key_iv: "9fcb67d11670fbb4cc0133cefb7aa4db"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.024983 7047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:35.027254 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.028479 7047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:35.028604 7047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "2834c08228574d1db3d9135a437af4ed"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "5b575e49ee91d0f3520bffb53281df6c"
server_key_iv: "9fcb67d11670fbb4cc0133cefb7aa4db"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.028707 7047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:35.046394 7047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:35.049182 7047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:35.049393 7047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:35.049949 7047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:35.050753 7047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:35.050805 7047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.050868 7047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:35.050899 7047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.059962 7047 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:43979
I20260620 19:00:35.060051 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:43979 every 8 connection(s)
I20260620 19:00:35.060552 7047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:35.066934 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7047
I20260620 19:00:35.067029 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:35.067343 2882 external_mini_cluster.cc:1428] Setting key 717d7463c4bbfad97821d59f18abf546
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:35.074445 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.062294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:58079 (local address 127.2.208.190:43073)
0620 19:00:35.062427 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.062431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.063210 (+ 779us) server_negotiation.cc:407] Connection header received
0620 19:00:35.064497 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.064500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.064547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.064620 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:35.065680 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.066219 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.067479 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.067732 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.067867 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.070536 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:35.070539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:35.070540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:35.070563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:35.072201 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.072720 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.072726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.072731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.072829 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.073144 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.073150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.073154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.073418 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:35.073527 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.074019 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.074181 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:00:35.075076 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.062614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43073 (local address 127.2.208.129:58079)
0620 19:00:35.063049 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.063090 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.064306 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.064747 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.064753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.065097 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:35.065557 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.065570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.066343 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.066346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.067310 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.067336 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.067905 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.067909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.068110 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.068693 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:35.068709 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:35.070358 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:35.072344 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.072348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.072360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.072618 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.072928 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.072931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.072932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.073064 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.073527 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:35.073532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:35.073771 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.073928 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.074173 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260620 19:00:35.076386 7175 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43073
I20260620 19:00:35.076629 7175 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:35.077066 7175 heartbeater.cc:511] Master 127.2.208.190:43073 requested a full tablet report, sending...
I20260620 19:00:35.078547 6984 ts_manager.cc:194] Registered new tserver with Master: 2834c08228574d1db3d9135a437af4ed (127.2.208.129:43979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:00:35.079919 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:58079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:35.093683 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:35.202128 7180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:35.202373 7180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:35.202454 7180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:35.206570 7180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:35.206625 7180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:35.206730 7180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:35.211752 7180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:35.212733 7180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:35.213770 7180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:35.222777 7185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.222905 7188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.222797 7186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.222811 7180 server_base.cc:1034] running on GCE node
I20260620 19:00:35.223451 7180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:35.224040 7180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:35.225219 7180 hybrid_clock.cc:648] HybridClock initialized: now 1781982035225194 us; error 41 us; skew 500 ppm
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:35.227916 7180 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:35.228606 7180 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:35.228667 7180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:35.228878 7180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:35.230893 7180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "88345701dfe543029a897975559acc5a"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "35ff215fa02c7f2fbebe7ac08505fbba"
server_key_iv: "e95729a6ab4dd1705d55e029944a6d21"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.231527 7180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "88345701dfe543029a897975559acc5a"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "35ff215fa02c7f2fbebe7ac08505fbba"
server_key_iv: "e95729a6ab4dd1705d55e029944a6d21"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.235265 7180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:35.237558 7194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.238744 7180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:35.238871 7180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "88345701dfe543029a897975559acc5a"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "35ff215fa02c7f2fbebe7ac08505fbba"
server_key_iv: "e95729a6ab4dd1705d55e029944a6d21"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.238976 7180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:35.250268 7180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:35.252969 7180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:35.253158 7180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:35.253685 7180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:35.254536 7180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:35.254606 7180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.254675 7180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:35.254719 7180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.263181 7180 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:46227
I20260620 19:00:35.263196 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:46227 every 8 connection(s)
I20260620 19:00:35.263798 7180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:35.269932 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7180
I20260620 19:00:35.270074 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:35.270377 2882 external_mini_cluster.cc:1428] Setting key 1fd50b758a065505949450eaaf2fd190
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:35.277973 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.265443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:57841 (local address 127.2.208.190:43073)
0620 19:00:35.265600 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.265605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.266453 (+ 848us) server_negotiation.cc:407] Connection header received
0620 19:00:35.267729 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.267731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.267783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.267857 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:35.268942 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.269465 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.270991 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.271221 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.271343 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.274070 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:35.274073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:35.274075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:35.274097 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:35.275864 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.276333 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.276339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.276344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.276441 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.276756 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.276760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.276761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.277025 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:35.277146 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.277600 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.277735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
I20260620 19:00:35.278602 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.265874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43073 (local address 127.2.208.130:57841)
0620 19:00:35.266305 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.266344 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.267543 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.268009 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.268015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.268355 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:35.268818 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.268830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.269567 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.269569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.270858 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.270867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.271405 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.271410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.271572 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.272110 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:35.272134 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:35.273901 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:35.275981 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.275985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.275996 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.276240 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.276540 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.276542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.276544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.276665 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.277129 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:35.277134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:35.277356 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.277514 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.277748 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260620 19:00:35.279738 7308 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43073
I20260620 19:00:35.279985 7308 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:35.280411 7308 heartbeater.cc:511] Master 127.2.208.190:43073 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:00:35.281392 6984 ts_manager.cc:194] Registered new tserver with Master: 88345701dfe543029a897975559acc5a (127.2.208.130:46227)
I20260620 19:00:35.281981 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:57841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:35.296452 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:35.405392 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:35.405736 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:35.405786 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:35.410650 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:35.410758 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:35.410856 7313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:35.417747 7313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:35.418963 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:35.420053 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:35.428910 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.428947 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.428932 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.429061 7313 server_base.cc:1034] running on GCE node
I20260620 19:00:35.429680 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:35.430312 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:35.431427 7313 hybrid_clock.cc:648] HybridClock initialized: now 1781982035431408 us; error 27 us; skew 500 ppm
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:35.434268 7313 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:35.434926 7313 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:35.434983 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:35.435138 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:35.437112 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "b86b6afc1bde7d69f8661ad32be65c94"
server_key_iv: "cd8e7cb016463fe8f317647604032d41"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.437608 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "b86b6afc1bde7d69f8661ad32be65c94"
server_key_iv: "cd8e7cb016463fe8f317647604032d41"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.441080 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:35.443331 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.444528 7313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:35.444641 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "b86b6afc1bde7d69f8661ad32be65c94"
server_key_iv: "cd8e7cb016463fe8f317647604032d41"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.444741 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:35.455746 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:35.458307 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:35.458490 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:35.459040 7313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:35.459861 7313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:35.459904 7313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.459967 7313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:35.460002 7313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.468775 7313 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:41069
I20260620 19:00:35.468794 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:41069 every 8 connection(s)
I20260620 19:00:35.469447 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:35.472452 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7313
I20260620 19:00:35.472565 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:35.472927 2882 external_mini_cluster.cc:1428] Setting key 924140d631f45743d24c30f901cc76be
Jun 20 19:00:35 dist-test-slave-tsm4 krb5kdc[6941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982035, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:35.482649 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:38517 (local address 127.2.208.190:43073)
0620 19:00:35.471362 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.471366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.472214 (+ 848us) server_negotiation.cc:407] Connection header received
0620 19:00:35.473839 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.473845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.473922 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.474008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:35.475477 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.475989 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.477043 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.477239 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.477315 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.479561 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:35.479564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:35.479566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:35.479587 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:35.481040 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.481457 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.481460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.481461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.481529 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.481804 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.481806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.481807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.481974 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:35.482043 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.482451 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.482534 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 19:00:35.483520 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.471624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43073 (local address 127.2.208.131:38517)
0620 19:00:35.472062 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.472104 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.473601 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.474195 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.474205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.474707 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:35.475298 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.475311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.476084 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.476086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.476940 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.476948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.477359 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.477362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.477493 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.477993 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:35.478008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:35.479410 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:35.481128 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.481132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.481142 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.481369 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.481610 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:35.481613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:35.481614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:35.481726 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:35.482052 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:35.482056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:35.482258 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.482393 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.482613 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260620 19:00:35.484642 7441 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43073
I20260620 19:00:35.484879 7441 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:35.485318 7441 heartbeater.cc:511] Master 127.2.208.190:43073 requested a full tablet report, sending...
I20260620 19:00:35.486361 6984 ts_manager.cc:194] Registered new tserver with Master: 79a66a21f4f64c059a93ae0d5a5f7cb5 (127.2.208.131:41069)
I20260620 19:00:35.486714 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:00:35.487082 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:38517
I20260620 19:00:35.495218 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.488833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.2.208.190:43073)
0620 19:00:35.488946 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.488949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.489128 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:00:35.489226 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.489228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.489262 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.489324 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:35.489912 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.490384 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.491147 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.491340 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.491473 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.492618 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:35.492620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:35.492622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:35.492641 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:35.493929 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.494285 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.494287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.494288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.494353 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:35.494587 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:35.494589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:35.494590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:35.494776 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:35.494873 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.495039 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.495116 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:00:35.497705 6984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:35.500677 6984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:35.514950 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.509387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:46227 (local address 127.0.0.1:53192)
0620 19:00:35.510149 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.510162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.510250 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.510921 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.510927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.510947 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:35.511048 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.511055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.512402 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.512407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.513517 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.513526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514383 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.514386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.514496 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.514531 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.514662 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.514792 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260620 19:00:35.514953 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.509577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.2.208.130:46227)
0620 19:00:35.510520 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.510531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.510552 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:00:35.510619 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.510624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.510791 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.510928 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:35.511142 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.512250 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.513671 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514225 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.514375 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.514529 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.514703 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.514795 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":96,"threads_started":1}
I20260620 19:00:35.515115 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.508862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:43979 (local address 127.0.0.1:40602)
0620 19:00:35.509096 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.509111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.509213 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.510227 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.510233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.510251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:35.510414 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.510421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.512371 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.512375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.513601 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.513608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514711 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.514716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.514832 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.514853 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.514900 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.514972 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":97,"thread_start_us":104,"threads_started":1}
I20260620 19:00:35.515340 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.508666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:41069 (local address 127.0.0.1:42124)
0620 19:00:35.509025 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.509076 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.509209 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.510766 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.510768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.510782 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:35.510881 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.510890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.512095 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.512100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.513597 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.513608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514971 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.514974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.515061 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.515080 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.515147 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.515213 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":37,"thread_start_us":87,"threads_started":1}
I20260620 19:00:35.515755 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.509679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.2.208.131:41069)
0620 19:00:35.509953 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.509964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.509986 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:00:35.510060 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.510066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.510632 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.510786 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:35.511018 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.511971 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.513749 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514697 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.514785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.514918 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.515526 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.515610 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
I20260620 19:00:35.515755 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.509476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.2.208.129:43979)
0620 19:00:35.509780 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.509791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.509814 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:00:35.509896 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.509899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.510079 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.510221 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:35.510954 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.512216 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.513757 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.514577 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.515283 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.515475 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.515525 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.515610 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260620 19:00:35.518415 7109 tablet_service.cc:1467] Processing CreateTablet for tablet 2caa9ed10b824070a65e228331b05520 (DEFAULT_TABLE table=test-table [id=087cbcba997d43ec897eb5380a7922e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:35.518509 7375 tablet_service.cc:1467] Processing CreateTablet for tablet 2caa9ed10b824070a65e228331b05520 (DEFAULT_TABLE table=test-table [id=087cbcba997d43ec897eb5380a7922e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:35.518443 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 2caa9ed10b824070a65e228331b05520 (DEFAULT_TABLE table=test-table [id=087cbcba997d43ec897eb5380a7922e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:35.519470 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caa9ed10b824070a65e228331b05520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:35.519471 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caa9ed10b824070a65e228331b05520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:35.519512 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2caa9ed10b824070a65e228331b05520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:35.526958 7460 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Bootstrap starting.
I20260620 19:00:35.527999 7458 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Bootstrap starting.
I20260620 19:00:35.529661 7460 tablet_bootstrap.cc:654] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:35.530066 7458 tablet_bootstrap.cc:654] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:35.531081 7460 log.cc:824] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:35.531277 7458 log.cc:824] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:35.531649 7459 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Bootstrap starting.
I20260620 19:00:35.532990 7460 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: No bootstrap required, opened a new log
I20260620 19:00:35.533193 7460 ts_tablet_manager.cc:1397] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:00:35.533398 7458 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: No bootstrap required, opened a new log
I20260620 19:00:35.533558 7458 ts_tablet_manager.cc:1397] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:00:35.533607 7459 tablet_bootstrap.cc:654] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:35.534786 7459 log.cc:824] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:35.536504 7460 raft_consensus.cc:348] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.536747 7460 raft_consensus.cc:374] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:35.536778 7459 tablet_bootstrap.cc:492] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: No bootstrap required, opened a new log
I20260620 19:00:35.536827 7460 raft_consensus.cc:729] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2834c08228574d1db3d9135a437af4ed, State: Initialized, Role: FOLLOWER
I20260620 19:00:35.536958 7459 ts_tablet_manager.cc:1397] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:35.537281 7460 consensus_queue.cc:260] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.537312 7458 raft_consensus.cc:348] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.537559 7458 raft_consensus.cc:374] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:35.537678 7458 raft_consensus.cc:729] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a66a21f4f64c059a93ae0d5a5f7cb5, State: Initialized, Role: FOLLOWER
I20260620 19:00:35.537930 7175 heartbeater.cc:503] Master 127.2.208.190:43073 was elected leader, sending a full tablet report...
I20260620 19:00:35.538264 7460 ts_tablet_manager.cc:1428] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:00:35.539227 7458 consensus_queue.cc:260] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.539965 7441 heartbeater.cc:503] Master 127.2.208.190:43073 was elected leader, sending a full tablet report...
I20260620 19:00:35.540243 7458 ts_tablet_manager.cc:1428] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260620 19:00:35.540575 7459 raft_consensus.cc:348] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.540747 7459 raft_consensus.cc:374] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:35.540786 7459 raft_consensus.cc:729] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88345701dfe543029a897975559acc5a, State: Initialized, Role: FOLLOWER
I20260620 19:00:35.541162 7459 consensus_queue.cc:260] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.542148 7459 ts_tablet_manager.cc:1428] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:35.542330 7308 heartbeater.cc:503] Master 127.2.208.190:43073 was elected leader, sending a full tablet report...
W20260620 19:00:35.561625 7176 tablet.cc:2369] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:35.658849 7464 raft_consensus.cc:482] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:35.659096 7464 raft_consensus.cc:504] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.660238 7464 leader_election.cc:288] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79a66a21f4f64c059a93ae0d5a5f7cb5 (127.2.208.131:41069), 88345701dfe543029a897975559acc5a (127.2.208.130:46227)
I20260620 19:00:35.664556 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.660610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:42277 (local address 127.2.208.131:41069)
0620 19:00:35.660766 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.660771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.660918 (+ 147us) server_negotiation.cc:407] Connection header received
0620 19:00:35.661090 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.661094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.661144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.661205 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:35.661788 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.662394 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.663528 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.663997 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.664087 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.664125 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.664351 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.664428 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:00:35.664559 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.660557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:41069 (local address 127.2.208.129:42277)
0620 19:00:35.660823 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.660837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.660951 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.661359 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.661362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.661382 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:35.661525 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.661533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.662506 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.662509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.663416 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.663425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.664117 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.664121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.664224 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.664247 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.664321 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.664385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260620 19:00:35.664677 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.661124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:35007 (local address 127.2.208.130:46227)
0620 19:00:35.661236 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.661241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.661527 (+ 286us) server_negotiation.cc:407] Connection header received
0620 19:00:35.661625 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.661630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.661684 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.661760 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:35.662030 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.662553 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.663681 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.664107 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.664220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.664281 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.664486 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.664555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:35.664708 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.660957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:46227 (local address 127.2.208.129:35007)
0620 19:00:35.661392 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.661424 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:35.661532 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:35.661788 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:35.661791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:35.661808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:35.661918 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.661924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.662664 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.662667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.663559 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:35.663569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.664237 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.664240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:35.664357 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.664381 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:00:35.664455 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:35.664518 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260620 19:00:35.665282 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caa9ed10b824070a65e228331b05520" candidate_uuid: "2834c08228574d1db3d9135a437af4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88345701dfe543029a897975559acc5a" is_pre_election: true
I20260620 19:00:35.665273 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caa9ed10b824070a65e228331b05520" candidate_uuid: "2834c08228574d1db3d9135a437af4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" is_pre_election: true
I20260620 19:00:35.665567 7262 raft_consensus.cc:2454] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2834c08228574d1db3d9135a437af4ed in term 0.
I20260620 19:00:35.665583 7395 raft_consensus.cc:2454] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2834c08228574d1db3d9135a437af4ed in term 0.
I20260620 19:00:35.666026 7065 leader_election.cc:302] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2834c08228574d1db3d9135a437af4ed, 88345701dfe543029a897975559acc5a; no voters:
I20260620 19:00:35.666298 7464 raft_consensus.cc:2781] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:35.666389 7464 raft_consensus.cc:482] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:35.666450 7464 raft_consensus.cc:3037] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:35.667510 7464 raft_consensus.cc:504] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.667827 7464 leader_election.cc:288] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [CANDIDATE]: Term 1 election: Requested vote from peers 79a66a21f4f64c059a93ae0d5a5f7cb5 (127.2.208.131:41069), 88345701dfe543029a897975559acc5a (127.2.208.130:46227)
I20260620 19:00:35.668128 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caa9ed10b824070a65e228331b05520" candidate_uuid: "2834c08228574d1db3d9135a437af4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5"
I20260620 19:00:35.668236 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2caa9ed10b824070a65e228331b05520" candidate_uuid: "2834c08228574d1db3d9135a437af4ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88345701dfe543029a897975559acc5a"
I20260620 19:00:35.668269 7395 raft_consensus.cc:3037] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:35.668351 7262 raft_consensus.cc:3037] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:35.669528 7395 raft_consensus.cc:2454] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2834c08228574d1db3d9135a437af4ed in term 1.
I20260620 19:00:35.669528 7262 raft_consensus.cc:2454] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2834c08228574d1db3d9135a437af4ed in term 1.
I20260620 19:00:35.669812 7065 leader_election.cc:302] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2834c08228574d1db3d9135a437af4ed, 79a66a21f4f64c059a93ae0d5a5f7cb5; no voters:
I20260620 19:00:35.669955 7464 raft_consensus.cc:2781] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:35.670181 7464 raft_consensus.cc:686] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [term 1 LEADER]: Becoming Leader. State: Replica: 2834c08228574d1db3d9135a437af4ed, State: Running, Role: LEADER
I20260620 19:00:35.670495 7464 consensus_queue.cc:237] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } }
I20260620 19:00:35.673408 6985 catalog_manager.cc:5495] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 2834c08228574d1db3d9135a437af4ed (127.2.208.129). New cstate: current_term: 1 leader_uuid: "2834c08228574d1db3d9135a437af4ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2834c08228574d1db3d9135a437af4ed" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:35.706337 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.702590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.2.208.129:43979)
0620 19:00:35.702722 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.702726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.702743 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:35.702849 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.702851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.702901 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.703023 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:35.703267 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.703874 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.704908 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:35.705131 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:35.705239 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:35.705542 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:35.705959 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:35.706092 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.706132 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.706186 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:35.712109 7395 raft_consensus.cc:1261] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 1 FOLLOWER]: Refusing update from remote peer 2834c08228574d1db3d9135a437af4ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:35.712109 7262 raft_consensus.cc:1261] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 1 FOLLOWER]: Refusing update from remote peer 2834c08228574d1db3d9135a437af4ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:35.712811 7469 consensus_queue.cc:1035] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a66a21f4f64c059a93ae0d5a5f7cb5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:35.712982 7464 consensus_queue.cc:1035] T 2caa9ed10b824070a65e228331b05520 P 2834c08228574d1db3d9135a437af4ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "88345701dfe543029a897975559acc5a" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:35.721688 7474 mvcc.cc:205] Tried to move back new op lower bound from 7298998418269163520 to 7298998418108882944. Current Snapshot: MvccSnapshot[applied={T|T < 7298998418269163520}]
I20260620 19:00:35.728442 6984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:35.728595 6984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:35.730901 6984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d678f9412e42d89ce8fb398e85d40a: Sending DeleteTablet for 3 replicas of tablet 2caa9ed10b824070a65e228331b05520
I20260620 19:00:35.731663 7375 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caa9ed10b824070a65e228331b05520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:35 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:42124
I20260620 19:00:35.731684 7109 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caa9ed10b824070a65e228331b05520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:35 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:40602
I20260620 19:00:35.732197 7481 tablet_replica.cc:330] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: stopping tablet replica
I20260620 19:00:35.732452 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2caa9ed10b824070a65e228331b05520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:35 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:53192
I20260620 19:00:35.732550 7481 raft_consensus.cc:2229] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:35.732913 7483 tablet_replica.cc:330] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: stopping tablet replica
I20260620 19:00:35.732930 7481 raft_consensus.cc:2258] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:35.733026 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7047
I20260620 19:00:35.733574 7483 raft_consensus.cc:2229] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:35.733846 7483 raft_consensus.cc:2258] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:35.735404 7483 ts_tablet_manager.cc:1905] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:35.736014 7481 ts_tablet_manager.cc:1905] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:35.737821 7483 ts_tablet_manager.cc:1918] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:35.737932 7483 log.cc:1196] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/2caa9ed10b824070a65e228331b05520
I20260620 19:00:35.738204 7483 ts_tablet_manager.cc:1939] T 2caa9ed10b824070a65e228331b05520 P 88345701dfe543029a897975559acc5a: Deleting consensus metadata
I20260620 19:00:35.739310 6972 catalog_manager.cc:4841] TS 88345701dfe543029a897975559acc5a (127.2.208.130:46227): tablet 2caa9ed10b824070a65e228331b05520 (table test-table [id=087cbcba997d43ec897eb5380a7922e6]) successfully deleted
I20260620 19:00:35.739389 7481 ts_tablet_manager.cc:1918] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:35.739523 7481 log.cc:1196] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/2caa9ed10b824070a65e228331b05520
I20260620 19:00:35.739850 7481 ts_tablet_manager.cc:1939] T 2caa9ed10b824070a65e228331b05520 P 79a66a21f4f64c059a93ae0d5a5f7cb5: Deleting consensus metadata
I20260620 19:00:35.741043 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7180
W20260620 19:00:35.741144 6970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:43979 (error 108)
I20260620 19:00:35.742340 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.742078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:43979 (local address 127.0.0.1:40628)
0620 19:00:35.742191 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:35.742269 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:00:35.742568 6970 catalog_manager.cc:4584] TS 2834c08228574d1db3d9135a437af4ed (127.2.208.129:43979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2caa9ed10b824070a65e228331b05520: Network error: Client connection negotiation failed: client connection to 127.2.208.129:43979: connect: Connection refused (error 111)
I20260620 19:00:35.743810 6972 catalog_manager.cc:4841] TS 79a66a21f4f64c059a93ae0d5a5f7cb5 (127.2.208.131:41069): tablet 2caa9ed10b824070a65e228331b05520 (table test-table [id=087cbcba997d43ec897eb5380a7922e6]) successfully deleted
I20260620 19:00:35.747752 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7313
I20260620 19:00:35.753162 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 6954
2026-06-20T19:00:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5071 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:35Z Disabled control of system clock
I20260620 19:00:35.782524 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34987
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:00:35.889030 7488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:35.889335 7488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:35.889374 7488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:35.893369 7488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:35.893424 7488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:35.893438 7488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:35.893455 7488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:35.893469 7488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:35.898305 7488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:34987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:35.899320 7488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:35.900652 7488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:35.908607 7493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.908672 7488 server_base.cc:1034] running on GCE node
W20260620 19:00:35.908640 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:35.908650 7494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:35.909420 7488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:35.910516 7488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:35.911672 7488 hybrid_clock.cc:648] HybridClock initialized: now 1781982035911637 us; error 46 us; skew 500 ppm
I20260620 19:00:35.913524 7488 webserver.cc:460] Webserver started at http://127.2.208.190:43765/ using document root <none> and password file <none>
I20260620 19:00:35.914103 7488 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:35.914162 7488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:35.914490 7488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:35.916602 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "d22212e7380c42d38e6754d63bc53176"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "d278f23dc2efc6d8ac7f3ef0a7f8022e"
server_key_iv: "0a34fe96dbd05961b3a39818e9b05141"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.917384 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "d22212e7380c42d38e6754d63bc53176"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "d278f23dc2efc6d8ac7f3ef0a7f8022e"
server_key_iv: "0a34fe96dbd05961b3a39818e9b05141"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.921088 7488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:35.923810 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:35.924986 7488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:35.925123 7488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "d22212e7380c42d38e6754d63bc53176"
format_stamp: "Formatted at 2026-06-20 19:00:35 on dist-test-slave-tsm4"
server_key: "d278f23dc2efc6d8ac7f3ef0a7f8022e"
server_key_iv: "0a34fe96dbd05961b3a39818e9b05141"
server_key_version: "encryptionkey@0"
I20260620 19:00:35.925251 7488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:35.929800 7488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:35.930421 7488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:35.930599 7488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:35.937247 7554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:34987 every 8 connection(s)
I20260620 19:00:35.937247 7488 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:34987
I20260620 19:00:35.938234 7488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:35.941378 7555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:35.946699 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: Bootstrap starting.
I20260620 19:00:35.947831 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7488
I20260620 19:00:35.947952 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:35.948202 2882 external_mini_cluster.cc:1428] Setting key f852d817e8c5ecf2865514da8dd22804
I20260620 19:00:35.949437 7555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:35.950147 7555 log.cc:824] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:35.952322 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: No bootstrap required, opened a new log
I20260620 19:00:35.953477 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:35.949347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.2.208.190:34987)
0620 19:00:35.950003 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:35.950019 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:35.950054 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:00:35.950823 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:35.950847 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:35.951153 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:35.951475 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:35.952238 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:35.952243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:35.952254 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:35.952263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:35.952312 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:35.952327 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:35.952460 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:35.952569 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:00:35.952835 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":163,"threads_started":1}
I20260620 19:00:35.956244 7555 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } }
I20260620 19:00:35.956444 7555 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:35.956519 7555 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22212e7380c42d38e6754d63bc53176, State: Initialized, Role: FOLLOWER
I20260620 19:00:35.957003 7555 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } }
I20260620 19:00:35.957144 7555 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:35.957211 7555 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:35.957312 7555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:35.958262 7555 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } }
I20260620 19:00:35.958632 7555 leader_election.cc:302] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22212e7380c42d38e6754d63bc53176; no voters:
I20260620 19:00:35.958956 7555 leader_election.cc:288] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:35.959060 7560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:35.959280 7560 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [term 1 LEADER]: Becoming Leader. State: Replica: d22212e7380c42d38e6754d63bc53176, State: Running, Role: LEADER
I20260620 19:00:35.959631 7560 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } }
I20260620 19:00:35.960173 7555 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:35.961340 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22212e7380c42d38e6754d63bc53176. Latest consensus state: current_term: 1 leader_uuid: "d22212e7380c42d38e6754d63bc53176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } } }
I20260620 19:00:35.961488 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:35.961614 7561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22212e7380c42d38e6754d63bc53176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22212e7380c42d38e6754d63bc53176" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34987 } } }
I20260620 19:00:35.961741 7561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176 [sys.catalog]: This master's current role is: LEADER
W20260620 19:00:35.964656 7575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:35.964751 7575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:35.964838 7569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:35.965762 7569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:35.970795 7569 catalog_manager.cc:1347] Generated new cluster ID: 96d9231aa3ec41988d49d8a7f0d4d13f
I20260620 19:00:35.970868 7569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:35.981607 7569 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:35.981886 7569 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:35.989923 7569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: Generated new TSK 0
I20260620 19:00:35.990669 7569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:00:36.000638 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--builtin_ntp_servers=127.2.208.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:00:36.110989 7579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:36.111281 7579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:36.111406 7579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:36.115780 7579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:36.115855 7579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:36.115940 7579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:36.120710 7579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:36.121778 7579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:36.123127 7579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:36.132958 7587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:36.133141 7579 server_base.cc:1034] running on GCE node
W20260620 19:00:36.133013 7584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:36.132953 7585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:36.133670 7579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:36.134338 7579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:36.135509 7579 hybrid_clock.cc:648] HybridClock initialized: now 1781982036135490 us; error 29 us; skew 500 ppm
I20260620 19:00:36.137421 7579 webserver.cc:460] Webserver started at http://127.2.208.129:42949/ using document root <none> and password file <none>
I20260620 19:00:36.138007 7579 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:36.138068 7579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:36.138340 7579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:36.140430 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "956810b1ee824e449e240267f2d9fe39"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "caf2835067a0f9e02d06c0ff2de08a3a"
server_key_iv: "f1d90594adab418a9e2da4d572a27240"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.140995 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "956810b1ee824e449e240267f2d9fe39"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "caf2835067a0f9e02d06c0ff2de08a3a"
server_key_iv: "f1d90594adab418a9e2da4d572a27240"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.144632 7579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:36.146955 7593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.148301 7579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:36.148438 7579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "956810b1ee824e449e240267f2d9fe39"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "caf2835067a0f9e02d06c0ff2de08a3a"
server_key_iv: "f1d90594adab418a9e2da4d572a27240"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.148547 7579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:36.152911 7579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:36.153481 7579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:36.153652 7579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:36.154150 7579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:36.155009 7579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:36.155079 7579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.155154 7579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:36.155205 7579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.164330 7579 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:33943
I20260620 19:00:36.164353 7706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:33943 every 8 connection(s)
I20260620 19:00:36.165256 7579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:36.166690 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7579
I20260620 19:00:36.166782 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:36.167095 2882 external_mini_cluster.cc:1428] Setting key e0d8a97a4d8ad3ca072cead507caa010
I20260620 19:00:36.168927 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--builtin_ntp_servers=127.2.208.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:00:36.171451 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.167002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:33959 (local address 127.2.208.190:34987)
0620 19:00:36.167194 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.167199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.168405 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 19:00:36.169761 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.169765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.169804 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.169883 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.170961 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.170964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.170967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.170979 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.171000 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.171009 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.171103 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.171255 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.171300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:00:36.172358 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.167477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34987 (local address 127.2.208.129:33959)
0620 19:00:36.168240 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.168302 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.169561 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.170041 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.170048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.170133 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.170721 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.170723 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.170764 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.170829 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.171100 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.171109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.171117 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.171296 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.171549 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":168,"threads_started":1}
I20260620 19:00:36.173522 7707 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34987
I20260620 19:00:36.173739 7707 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:36.174136 7707 heartbeater.cc:511] Master 127.2.208.190:34987 requested a full tablet report, sending...
I20260620 19:00:36.175777 7519 ts_manager.cc:194] Registered new tserver with Master: 956810b1ee824e449e240267f2d9fe39 (127.2.208.129:33943)
W20260620 19:00:36.277889 7710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:36.278162 7710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:36.278201 7710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:36.282272 7710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:36.282327 7710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:36.282383 7710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:36.287334 7710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:36.288352 7710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:36.289660 7710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:36.299041 7718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:36.299042 7716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:36.299041 7715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:36.299333 7710 server_base.cc:1034] running on GCE node
I20260620 19:00:36.299786 7710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:36.300354 7710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:36.301518 7710 hybrid_clock.cc:648] HybridClock initialized: now 1781982036301504 us; error 26 us; skew 500 ppm
I20260620 19:00:36.303277 7710 webserver.cc:460] Webserver started at http://127.2.208.130:38893/ using document root <none> and password file <none>
I20260620 19:00:36.303908 7710 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:36.303970 7710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:36.304200 7710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:36.306329 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "37aa184e760e4c75a354304494bac9b1"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "adec60db226eeb688c88cc915e7f96a1"
server_key_iv: "88c47812e2e72cbfbacb5dd37d80b1fe"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.306878 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "37aa184e760e4c75a354304494bac9b1"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "adec60db226eeb688c88cc915e7f96a1"
server_key_iv: "88c47812e2e72cbfbacb5dd37d80b1fe"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.310451 7710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:36.313184 7724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.314436 7710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:36.314577 7710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "37aa184e760e4c75a354304494bac9b1"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "adec60db226eeb688c88cc915e7f96a1"
server_key_iv: "88c47812e2e72cbfbacb5dd37d80b1fe"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.314699 7710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:36.319078 7710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:36.319690 7710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:36.319869 7710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:36.320367 7710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:36.321236 7710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:36.321305 7710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.321370 7710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:36.321424 7710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.329778 7710 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44025
I20260620 19:00:36.329823 7837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44025 every 8 connection(s)
I20260620 19:00:36.330667 7710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:36.335290 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7710
I20260620 19:00:36.335435 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:36.335758 2882 external_mini_cluster.cc:1428] Setting key 87c64af10844c142a6a2e6bb7455bc8b
I20260620 19:00:36.336534 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.332470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59041 (local address 127.2.208.190:34987)
0620 19:00:36.332603 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.332606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.333627 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 19:00:36.334855 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.334858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.334893 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.334964 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.336053 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.336056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.336059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.336070 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.336091 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.336107 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.336197 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.336345 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.336387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:36.337433 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.332871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34987 (local address 127.2.208.130:59041)
0620 19:00:36.333481 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.333519 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.334673 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.335084 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.335090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.335170 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.335767 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.335768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.335827 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.335912 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.336192 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.336199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.336206 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.336382 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.336623 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260620 19:00:36.337826 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--builtin_ntp_servers=127.2.208.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:00:36.338606 7838 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34987
I20260620 19:00:36.338835 7838 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:36.339280 7838 heartbeater.cc:511] Master 127.2.208.190:34987 requested a full tablet report, sending...
I20260620 19:00:36.340327 7519 ts_manager.cc:194] Registered new tserver with Master: 37aa184e760e4c75a354304494bac9b1 (127.2.208.130:44025)
W20260620 19:00:36.447329 7841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:36.447641 7841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:36.447680 7841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:36.451660 7841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:36.451710 7841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:36.451766 7841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:36.456517 7841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:36.457479 7841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:36.458799 7841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:36.468688 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:36.468688 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:36.468748 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:36.468724 7841 server_base.cc:1034] running on GCE node
I20260620 19:00:36.469367 7841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:36.469995 7841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:36.471174 7841 hybrid_clock.cc:648] HybridClock initialized: now 1781982036471141 us; error 46 us; skew 500 ppm
I20260620 19:00:36.473160 7841 webserver.cc:460] Webserver started at http://127.2.208.131:35217/ using document root <none> and password file <none>
I20260620 19:00:36.473789 7841 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:36.473875 7841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:36.474092 7841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:36.476187 7841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "bba162ab61f641bfadee3954ae9ec9c6"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "55bb2faeb2af6a3eeb79c67be9153d8d"
server_key_iv: "942b602a667d2067a4b289261638b84e"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.476754 7841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "bba162ab61f641bfadee3954ae9ec9c6"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "55bb2faeb2af6a3eeb79c67be9153d8d"
server_key_iv: "942b602a667d2067a4b289261638b84e"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.480357 7841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:36.482566 7855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.483876 7841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:36.483990 7841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "bba162ab61f641bfadee3954ae9ec9c6"
format_stamp: "Formatted at 2026-06-20 19:00:36 on dist-test-slave-tsm4"
server_key: "55bb2faeb2af6a3eeb79c67be9153d8d"
server_key_iv: "942b602a667d2067a4b289261638b84e"
server_key_version: "encryptionkey@0"
I20260620 19:00:36.484091 7841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:36.488481 7841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:36.489044 7841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:36.489270 7841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:36.489879 7841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:36.490739 7841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:36.490783 7841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.490820 7841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:36.490835 7841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:36.499533 7841 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:43959
I20260620 19:00:36.499555 7968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:43959 every 8 connection(s)
I20260620 19:00:36.500406 7841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:36.504252 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 7841
I20260620 19:00:36.504391 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:36.504714 2882 external_mini_cluster.cc:1428] Setting key 7f91058498854014c153ec51c33f17a7
I20260620 19:00:36.506801 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.502193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:41211 (local address 127.2.208.190:34987)
0620 19:00:36.502325 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.502328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.503431 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 19:00:36.505059 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.505062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.505107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.505190 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.506371 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.506375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.506377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.506388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.506409 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.506419 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.506508 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.506631 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.506675 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:00:36.507730 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.502559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34987 (local address 127.2.208.131:41211)
0620 19:00:36.503221 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.503260 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.504802 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.505372 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.505385 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.505500 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.506118 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.506121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.506168 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.506232 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.506502 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.506508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.506512 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.506672 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.506925 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260620 19:00:36.508627 7969 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34987
I20260620 19:00:36.508823 7969 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:36.509227 7969 heartbeater.cc:511] Master 127.2.208.190:34987 requested a full tablet report, sending...
I20260620 19:00:36.510106 7519 ts_manager.cc:194] Registered new tserver with Master: bba162ab61f641bfadee3954ae9ec9c6 (127.2.208.131:43959)
I20260620 19:00:36.518299 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:00:36.521546 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.520226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.2.208.190:34987)
0620 19:00:36.520373 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.520377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.520449 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:00:36.520542 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.520544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.520572 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.520634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.521201 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.521204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.521206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.521215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.521231 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.521238 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.521311 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.521370 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.521398 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 19:00:36.523932 7519 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:36.526715 7519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:36.539935 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.537114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43959 (local address 127.0.0.1:40724)
0620 19:00:36.537833 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.537844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.537922 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.538363 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.538372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.538450 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.539069 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.539073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.539094 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.539168 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.539593 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.539601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.539606 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.539704 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.539785 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":108,"threads_started":1}
I20260620 19:00:36.540002 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.536515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.2.208.129:33943)
0620 19:00:36.536855 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.536866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.537617 (+ 751us) server_negotiation.cc:407] Connection header received
0620 19:00:36.537801 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.537807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.538032 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.538158 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.539373 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.539403 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.539422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.539433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.539466 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.539489 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.539552 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.539782 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.539854 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":134,"threads_started":1}
I20260620 19:00:36.539940 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.536406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33943 (local address 127.0.0.1:45396)
0620 19:00:36.537471 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.537530 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.537683 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.538824 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.538833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.538847 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.539170 (+ 323us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.539171 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.539178 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.539234 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.539610 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.539612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.539613 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.539749 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.539785 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":65,"threads_started":1}
I20260620 19:00:36.540505 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.536466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44025 (local address 127.0.0.1:34056)
0620 19:00:36.537565 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.537579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.537674 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.538456 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.538458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.538470 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.539431 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.539433 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.539442 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.539467 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.540285 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.540288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.540291 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.540366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.540398 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":105,"threads_started":1}
I20260620 19:00:36.540591 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.537274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.2.208.131:43959)
0620 19:00:36.537666 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.537676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.538000 (+ 324us) server_negotiation.cc:407] Connection header received
0620 19:00:36.538051 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.538056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.538216 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.538333 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.539316 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.539323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.539334 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.539345 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.539411 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.539452 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.539555 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.540363 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.540436 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
I20260620 19:00:36.540899 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.537518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.2.208.130:44025)
0620 19:00:36.538054 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.538066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.538088 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:00:36.538140 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.538145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.538321 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.539273 (+ 952us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.540072 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.540080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.540089 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.540099 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.540134 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.540168 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.540672 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.540718 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.540782 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":133,"threads_started":1}
I20260620 19:00:36.543152 7641 tablet_service.cc:1467] Processing CreateTablet for tablet 2899099cca134465b9b47c2b9114c9f9 (DEFAULT_TABLE table=test-table [id=58bb7dacd19443df849549f0dbc42ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:36.543586 7772 tablet_service.cc:1467] Processing CreateTablet for tablet 2899099cca134465b9b47c2b9114c9f9 (DEFAULT_TABLE table=test-table [id=58bb7dacd19443df849549f0dbc42ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:36.544328 7641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2899099cca134465b9b47c2b9114c9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:36.544582 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2899099cca134465b9b47c2b9114c9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:36.545214 7903 tablet_service.cc:1467] Processing CreateTablet for tablet 2899099cca134465b9b47c2b9114c9f9 (DEFAULT_TABLE table=test-table [id=58bb7dacd19443df849549f0dbc42ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:36.546118 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2899099cca134465b9b47c2b9114c9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:36.550258 7986 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Bootstrap starting.
I20260620 19:00:36.551843 7987 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Bootstrap starting.
I20260620 19:00:36.553097 7986 tablet_bootstrap.cc:654] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:36.553972 7987 tablet_bootstrap.cc:654] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:36.553997 7986 log.cc:824] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:36.554659 7988 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Bootstrap starting.
I20260620 19:00:36.554783 7987 log.cc:824] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:36.555938 7986 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: No bootstrap required, opened a new log
I20260620 19:00:36.556146 7986 ts_tablet_manager.cc:1397] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 19:00:36.556640 7987 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: No bootstrap required, opened a new log
I20260620 19:00:36.556671 7988 tablet_bootstrap.cc:654] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:36.556828 7987 ts_tablet_manager.cc:1397] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:36.557451 7988 log.cc:824] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:36.559803 7988 tablet_bootstrap.cc:492] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: No bootstrap required, opened a new log
I20260620 19:00:36.559993 7988 ts_tablet_manager.cc:1397] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 19:00:36.560176 7986 raft_consensus.cc:348] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.560406 7986 raft_consensus.cc:374] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:36.560485 7986 raft_consensus.cc:729] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aa184e760e4c75a354304494bac9b1, State: Initialized, Role: FOLLOWER
I20260620 19:00:36.561024 7986 consensus_queue.cc:260] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.561640 7838 heartbeater.cc:503] Master 127.2.208.190:34987 was elected leader, sending a full tablet report...
I20260620 19:00:36.561980 7986 ts_tablet_manager.cc:1428] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:00:36.561353 7987 raft_consensus.cc:348] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.562844 7987 raft_consensus.cc:374] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:36.562922 7987 raft_consensus.cc:729] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956810b1ee824e449e240267f2d9fe39, State: Initialized, Role: FOLLOWER
I20260620 19:00:36.563350 7987 consensus_queue.cc:260] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.564043 7707 heartbeater.cc:503] Master 127.2.208.190:34987 was elected leader, sending a full tablet report...
I20260620 19:00:36.564139 7988 raft_consensus.cc:348] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.564356 7987 ts_tablet_manager.cc:1428] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260620 19:00:36.564394 7988 raft_consensus.cc:374] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:36.564446 7988 raft_consensus.cc:729] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba162ab61f641bfadee3954ae9ec9c6, State: Initialized, Role: FOLLOWER
I20260620 19:00:36.564813 7988 consensus_queue.cc:260] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.565568 7988 ts_tablet_manager.cc:1428] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:00:36.565883 7969 heartbeater.cc:503] Master 127.2.208.190:34987 was elected leader, sending a full tablet report...
W20260620 19:00:36.581563 7839 tablet.cc:2369] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:36.666368 7708 tablet.cc:2369] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:36.751544 7970 tablet.cc:2369] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:36.829775 7994 raft_consensus.cc:482] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:36.829984 7994 raft_consensus.cc:504] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.831179 7994 leader_election.cc:288] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 956810b1ee824e449e240267f2d9fe39 (127.2.208.129:33943), 37aa184e760e4c75a354304494bac9b1 (127.2.208.130:44025)
I20260620 19:00:36.833042 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.831383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33943 (local address 127.2.208.131:49313)
0620 19:00:36.831546 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.831561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.831641 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.831919 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.831921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.831937 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.832483 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.832484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.832493 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.832514 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.832781 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.832784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.832787 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.832867 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.832901 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 19:00:36.833098 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.831572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:49313 (local address 127.2.208.129:33943)
0620 19:00:36.831713 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.831719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.831734 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:36.831786 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.831787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.831815 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.831902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.832636 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.832641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.832648 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.832659 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.832675 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.832682 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.832792 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.832904 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.832952 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:00:36.833263 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.831452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44025 (local address 127.2.208.131:36607)
0620 19:00:36.831858 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.831869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:36.831976 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:36.832266 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:36.832270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:36.832287 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:36.832754 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:00:36.832755 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:36.832764 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:00:36.832785 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:36.833009 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:36.833012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:36.833015 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:00:36.833101 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:00:36.833135 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260620 19:00:36.833284 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.831563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:36607 (local address 127.2.208.130:44025)
0620 19:00:36.831686 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.831691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.831953 (+ 262us) server_negotiation.cc:407] Connection header received
0620 19:00:36.832081 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.832085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.832147 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.832233 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.832890 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.832894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.832896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.832906 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.832925 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.832934 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.833007 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.833111 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.833149 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:00:36.833854 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2899099cca134465b9b47c2b9114c9f9" candidate_uuid: "bba162ab61f641bfadee3954ae9ec9c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956810b1ee824e449e240267f2d9fe39" is_pre_election: true
I20260620 19:00:36.833854 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2899099cca134465b9b47c2b9114c9f9" candidate_uuid: "bba162ab61f641bfadee3954ae9ec9c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aa184e760e4c75a354304494bac9b1" is_pre_election: true
I20260620 19:00:36.834134 7661 raft_consensus.cc:2454] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba162ab61f641bfadee3954ae9ec9c6 in term 0.
I20260620 19:00:36.834152 7792 raft_consensus.cc:2454] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba162ab61f641bfadee3954ae9ec9c6 in term 0.
I20260620 19:00:36.834592 7857 leader_election.cc:302] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aa184e760e4c75a354304494bac9b1, bba162ab61f641bfadee3954ae9ec9c6; no voters:
I20260620 19:00:36.834860 7994 raft_consensus.cc:2781] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:36.834940 7994 raft_consensus.cc:482] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:36.834971 7994 raft_consensus.cc:3037] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:36.836076 7994 raft_consensus.cc:504] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.836457 7994 leader_election.cc:288] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [CANDIDATE]: Term 1 election: Requested vote from peers 956810b1ee824e449e240267f2d9fe39 (127.2.208.129:33943), 37aa184e760e4c75a354304494bac9b1 (127.2.208.130:44025)
I20260620 19:00:36.836772 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2899099cca134465b9b47c2b9114c9f9" candidate_uuid: "bba162ab61f641bfadee3954ae9ec9c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956810b1ee824e449e240267f2d9fe39"
I20260620 19:00:36.836849 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2899099cca134465b9b47c2b9114c9f9" candidate_uuid: "bba162ab61f641bfadee3954ae9ec9c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aa184e760e4c75a354304494bac9b1"
I20260620 19:00:36.836915 7661 raft_consensus.cc:3037] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:36.836966 7792 raft_consensus.cc:3037] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:36.837931 7661 raft_consensus.cc:2454] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba162ab61f641bfadee3954ae9ec9c6 in term 1.
I20260620 19:00:36.837945 7792 raft_consensus.cc:2454] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba162ab61f641bfadee3954ae9ec9c6 in term 1.
I20260620 19:00:36.838236 7859 leader_election.cc:302] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 956810b1ee824e449e240267f2d9fe39, bba162ab61f641bfadee3954ae9ec9c6; no voters:
I20260620 19:00:36.838402 7994 raft_consensus.cc:2781] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:36.838622 7994 raft_consensus.cc:686] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 1 LEADER]: Becoming Leader. State: Replica: bba162ab61f641bfadee3954ae9ec9c6, State: Running, Role: LEADER
I20260620 19:00:36.838937 7994 consensus_queue.cc:237] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } }
I20260620 19:00:36.842061 7518 catalog_manager.cc:5495] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bba162ab61f641bfadee3954ae9ec9c6 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "bba162ab61f641bfadee3954ae9ec9c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba162ab61f641bfadee3954ae9ec9c6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43959 } health_report { overall_health: HEALTHY } } }
I20260620 19:00:36.901575 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.900280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.2.208.131:43959)
0620 19:00:36.900407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:36.900410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:36.900422 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:36.900492 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:36.900494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:36.900549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:36.900632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:36.901248 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:36.901251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:36.901254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:36.901264 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:36.901280 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:36.901288 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:36.901358 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:36.901440 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:00:36.901474 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:36.907292 7661 raft_consensus.cc:1261] T 2899099cca134465b9b47c2b9114c9f9 P 956810b1ee824e449e240267f2d9fe39 [term 1 FOLLOWER]: Refusing update from remote peer bba162ab61f641bfadee3954ae9ec9c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:36.907292 7792 raft_consensus.cc:1261] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 1 FOLLOWER]: Refusing update from remote peer bba162ab61f641bfadee3954ae9ec9c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:36.907994 7996 consensus_queue.cc:1035] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37aa184e760e4c75a354304494bac9b1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:36.908176 7994 consensus_queue.cc:1035] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956810b1ee824e449e240267f2d9fe39" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:36.915719 8001 mvcc.cc:205] Tried to move back new op lower bound from 7298998423164923904 to 7298998422894874624. Current Snapshot: MvccSnapshot[applied={T|T < 7298998423164923904}]
I20260620 19:00:36.917400 8000 mvcc.cc:205] Tried to move back new op lower bound from 7298998423164923904 to 7298998422894874624. Current Snapshot: MvccSnapshot[applied={T|T < 7298998423164923904}]
I20260620 19:00:36.922933 7518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:36.923106 7518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:36.925544 7518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d22212e7380c42d38e6754d63bc53176: Sending DeleteTablet for 3 replicas of tablet 2899099cca134465b9b47c2b9114c9f9
I20260620 19:00:36.926282 7641 tablet_service.cc:1514] Processing DeleteTablet for tablet 2899099cca134465b9b47c2b9114c9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:36 UTC) from {username='slave'} at 127.0.0.1:45396
I20260620 19:00:36.926350 7903 tablet_service.cc:1514] Processing DeleteTablet for tablet 2899099cca134465b9b47c2b9114c9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:36 UTC) from {username='slave'} at 127.0.0.1:40724
I20260620 19:00:36.926429 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet 2899099cca134465b9b47c2b9114c9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:36 UTC) from {username='slave'} at 127.0.0.1:34056
I20260620 19:00:36.926743 8008 tablet_replica.cc:330] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: stopping tablet replica
I20260620 19:00:36.926872 8010 tablet_replica.cc:330] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: stopping tablet replica
I20260620 19:00:36.927026 8008 raft_consensus.cc:2229] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:36.927098 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7579
I20260620 19:00:36.927161 8010 raft_consensus.cc:2229] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:36.927703 8008 raft_consensus.cc:2258] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:36.928601 8010 raft_consensus.cc:2258] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:36.929934 8008 ts_tablet_manager.cc:1905] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:36.930054 8010 ts_tablet_manager.cc:1905] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:36.932206 8010 ts_tablet_manager.cc:1918] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:36.932312 8010 log.cc:1196] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/2899099cca134465b9b47c2b9114c9f9
I20260620 19:00:36.932631 8010 ts_tablet_manager.cc:1939] T 2899099cca134465b9b47c2b9114c9f9 P 37aa184e760e4c75a354304494bac9b1: Deleting consensus metadata
I20260620 19:00:36.932601 8008 ts_tablet_manager.cc:1918] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:36.932722 8008 log.cc:1196] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/2899099cca134465b9b47c2b9114c9f9
I20260620 19:00:36.933116 8008 ts_tablet_manager.cc:1939] T 2899099cca134465b9b47c2b9114c9f9 P bba162ab61f641bfadee3954ae9ec9c6: Deleting consensus metadata
I20260620 19:00:36.933780 7504 catalog_manager.cc:4841] TS 37aa184e760e4c75a354304494bac9b1 (127.2.208.130:44025): tablet 2899099cca134465b9b47c2b9114c9f9 (table test-table [id=58bb7dacd19443df849549f0dbc42ff4]) successfully deleted
I20260620 19:00:36.934113 7504 catalog_manager.cc:4841] TS bba162ab61f641bfadee3954ae9ec9c6 (127.2.208.131:43959): tablet 2899099cca134465b9b47c2b9114c9f9 (table test-table [id=58bb7dacd19443df849549f0dbc42ff4]) successfully deleted
W20260620 19:00:36.939630 7506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:33943 (error 108)
I20260620 19:00:36.940881 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:36.940512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33943 (local address 127.0.0.1:45404)
0620 19:00:36.940713 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:36.940784 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 19:00:36.941028 7506 catalog_manager.cc:4584] TS 956810b1ee824e449e240267f2d9fe39 (127.2.208.129:33943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2899099cca134465b9b47c2b9114c9f9: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33943: connect: Connection refused (error 111)
I20260620 19:00:36.941469 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7710
I20260620 19:00:36.948097 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7841
I20260620 19:00:36.954402 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 7488
2026-06-20T19:00:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1200 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:00:36.975760 2882 mini_oidc.cc:152] Starting JWKS server
I20260620 19:00:36.977375 2882 webserver.cc:460] Webserver started at https://127.0.0.1:43407/ using document root <none> and password file <none>
I20260620 19:00:36.977550 2882 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:00:36.977808 2882 webserver.cc:460] Webserver started at http://127.0.0.1:36549/ using document root <none> and password file <none>
2026-06-20T19:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:36Z Disabled control of system clock
I20260620 19:00:36.986390 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39271
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39271
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/testchainca.cert with env {}
W20260620 19:00:37.092384 8017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:37.092674 8017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:37.092741 8017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:37.096917 8017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:37.096999 8017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:37.097026 8017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:37.097079 8017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:37.097115 8017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:00:37.097147 8017 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:00:37.097177 8017 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43407/jwks.json
I20260620 19:00:37.102562 8017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:37.103705 8017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:37.104702 8017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:37.112733 8022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:37.112802 8025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:37.112771 8023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:37.112772 8017 server_base.cc:1034] running on GCE node
I20260620 19:00:37.113415 8017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:37.114583 8017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:37.115758 8017 hybrid_clock.cc:648] HybridClock initialized: now 1781982037115731 us; error 40 us; skew 500 ppm
I20260620 19:00:37.121277 8017 webserver.cc:460] Webserver started at http://127.2.208.190:42441/ using document root <none> and password file <none>
I20260620 19:00:37.121874 8017 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:37.121937 8017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:37.122094 8017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:37.124195 8017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "1ec7e8b615444fbba33655aac9aa79e8"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "555cfc5936bfefe0008d26e843199e70"
server_key_iv: "7628bfa04d3158b28c3986f93ab22afe"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.124716 8017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "1ec7e8b615444fbba33655aac9aa79e8"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "555cfc5936bfefe0008d26e843199e70"
server_key_iv: "7628bfa04d3158b28c3986f93ab22afe"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.128201 8017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:37.130502 8031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:37.131759 8017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:37.131882 8017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "1ec7e8b615444fbba33655aac9aa79e8"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "555cfc5936bfefe0008d26e843199e70"
server_key_iv: "7628bfa04d3158b28c3986f93ab22afe"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.131985 8017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:37.145671 8017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:37.146288 8017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:37.146492 8017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:37.153398 8017 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39271
I20260620 19:00:37.153406 8083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39271 every 8 connection(s)
I20260620 19:00:37.154331 8017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:37.157358 8084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:37.161801 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8017
I20260620 19:00:37.161943 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:37.162215 2882 external_mini_cluster.cc:1428] Setting key 7f76d6731c95c5ca2aa70cc26933b45a
I20260620 19:00:37.163076 8084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8: Bootstrap starting.
I20260620 19:00:37.165365 8084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:37.166087 8084 log.cc:824] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:37.168026 8084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8: No bootstrap required, opened a new log
I20260620 19:00:37.171463 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.163561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.2.208.190:39271)
0620 19:00:37.164179 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.164193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.164243 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:00:37.164840 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.164855 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.165240 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.165564 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:37.165992 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.167170 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.168190 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.168654 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.168830 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.169684 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:37.169689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:37.169701 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:37.170069 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:37.170120 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:37.170140 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:37.170265 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:37.170443 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:00:37.170696 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260620 19:00:37.171877 8084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } }
I20260620 19:00:37.172143 8084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:37.172196 8084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec7e8b615444fbba33655aac9aa79e8, State: Initialized, Role: FOLLOWER
I20260620 19:00:37.172641 8084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } }
I20260620 19:00:37.172760 8084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:37.172843 8084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:37.172955 8084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:37.173981 8084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } }
I20260620 19:00:37.174371 8084 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec7e8b615444fbba33655aac9aa79e8; no voters:
I20260620 19:00:37.174420 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:00:37.174685 8084 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:37.174762 8089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:37.174940 8089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec7e8b615444fbba33655aac9aa79e8, State: Running, Role: LEADER
I20260620 19:00:37.175274 8089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } }
I20260620 19:00:37.175922 8084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:37.177151 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec7e8b615444fbba33655aac9aa79e8. Latest consensus state: current_term: 1 leader_uuid: "1ec7e8b615444fbba33655aac9aa79e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } } }
I20260620 19:00:37.177146 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec7e8b615444fbba33655aac9aa79e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7e8b615444fbba33655aac9aa79e8" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39271 } } }
I20260620 19:00:37.177299 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:37.177299 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:37.177749 8098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:37.180927 8098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:37.186661 8098 catalog_manager.cc:1347] Generated new cluster ID: a798ea182781444e8c88015343e72454
I20260620 19:00:37.186734 8098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:37.205878 8098 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:37.206777 8098 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:37.213142 8098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec7e8b615444fbba33655aac9aa79e8: Generated new TSK 0
I20260620 19:00:37.213792 8098 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:00:37.240200 8114 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:00:37.240406 8114 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:00:37.240962 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.224368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.2.208.190:39271)
0620 19:00:37.224494 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.224499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.224715 (+ 216us) server_negotiation.cc:407] Connection header received
0620 19:00:37.224953 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.224962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.225008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.225106 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:00:37.225385 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.225921 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.226772 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.226997 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.227103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.227459 (+ 356us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:00:37.240501 (+ 13042us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:00:37.240807 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 19:00:37.241046 8087 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:00:37.242077 8113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:37.224285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39271 (local address 127.0.0.1:43250)
0620 19:00:37.224626 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:37.224639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:37.224783 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:37.225133 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:37.225136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:37.225144 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:00:37.225276 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.225281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.226041 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.226047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.226650 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.226657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.227108 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.227112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.227203 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.227259 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:00:37.240734 (+ 13475us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:00:37.240954 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.208.190:39271: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.208.190:39271: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:00:37.242741 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8017
2026-06-20T19:00:37Z chronyd exiting
I20260620 19:00:37.280471 2882 test_util.cc:175] -----------------------------------------------
I20260620 19:00:37.280556 2882 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982008625501-2882-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:00:37.281562 2882 mini_oidc.cc:152] Starting JWKS server
I20260620 19:00:37.282395 2882 webserver.cc:460] Webserver started at https://127.0.0.1:32969/ using document root <none> and password file <none>
I20260620 19:00:37.282513 2882 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:00:37.282788 2882 webserver.cc:460] Webserver started at http://127.0.0.1:36603/ using document root <none> and password file <none>
2026-06-20T19:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:37Z Disabled control of system clock
I20260620 19:00:37.291654 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39263
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39263
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/testchainca.cert with env {}
W20260620 19:00:37.401247 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:37.401500 8120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:37.401541 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:37.405517 8120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:37.405572 8120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:37.405587 8120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:37.405632 8120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:37.405659 8120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:00:37.405694 8120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:00:37.405738 8120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32969/jwks.json
I20260620 19:00:37.411006 8120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:37.411984 8120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:37.412881 8120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:37.421506 8125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:37.421532 8128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:37.421541 8126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:37.421761 8120 server_base.cc:1034] running on GCE node
I20260620 19:00:37.422322 8120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:37.423424 8120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:37.424613 8120 hybrid_clock.cc:648] HybridClock initialized: now 1781982037424603 us; error 30 us; skew 500 ppm
I20260620 19:00:37.426357 8120 webserver.cc:460] Webserver started at http://127.2.208.190:36789/ using document root <none> and password file <none>
I20260620 19:00:37.426910 8120 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:37.426998 8120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:37.427222 8120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:37.429286 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "df32be4bb685457d92f11fb1b0c7809a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "ac0e932e6d65ca593eaebb9ff46f15a2"
server_key_iv: "5b49739b1a6c8692888a7251d15479cf"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.429836 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "df32be4bb685457d92f11fb1b0c7809a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "ac0e932e6d65ca593eaebb9ff46f15a2"
server_key_iv: "5b49739b1a6c8692888a7251d15479cf"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.433326 8120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:37.435628 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:37.436817 8120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:00:37.436952 8120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "df32be4bb685457d92f11fb1b0c7809a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "ac0e932e6d65ca593eaebb9ff46f15a2"
server_key_iv: "5b49739b1a6c8692888a7251d15479cf"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.437060 8120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:37.482988 8120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:37.483803 8120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:37.484015 8120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:37.490775 8120 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39263
I20260620 19:00:37.490789 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39263 every 8 connection(s)
I20260620 19:00:37.491809 8120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:37.494839 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:37.497138 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8120
I20260620 19:00:37.497237 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:37.497503 2882 external_mini_cluster.cc:1428] Setting key 8624b904474fe073148491b5de453f88
I20260620 19:00:37.500627 8187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: Bootstrap starting.
I20260620 19:00:37.503050 8187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:37.503845 8187 log.cc:824] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:37.505987 8187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: No bootstrap required, opened a new log
I20260620 19:00:37.506873 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.498859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.2.208.190:39263)
0620 19:00:37.499552 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.499567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.499602 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:00:37.500286 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.500304 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.500711 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.501025 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:37.501362 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.502304 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.503488 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.503964 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.504083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.504953 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:37.504959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:37.504974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:37.505440 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:37.505506 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:37.505531 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:37.505628 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:37.505745 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:00:37.506004 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260620 19:00:37.509657 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:00:37.510179 8187 raft_consensus.cc:348] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } }
I20260620 19:00:37.510434 8187 raft_consensus.cc:374] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:37.510504 8187 raft_consensus.cc:729] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df32be4bb685457d92f11fb1b0c7809a, State: Initialized, Role: FOLLOWER
I20260620 19:00:37.510936 8187 consensus_queue.cc:260] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } }
I20260620 19:00:37.511109 8187 raft_consensus.cc:388] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:37.511176 8187 raft_consensus.cc:482] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:37.511282 8187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:37.512176 8187 raft_consensus.cc:504] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } }
I20260620 19:00:37.512501 8187 leader_election.cc:302] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df32be4bb685457d92f11fb1b0c7809a; no voters:
I20260620 19:00:37.512785 8187 leader_election.cc:288] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:37.512923 8192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:37.513145 8192 raft_consensus.cc:686] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [term 1 LEADER]: Becoming Leader. State: Replica: df32be4bb685457d92f11fb1b0c7809a, State: Running, Role: LEADER
I20260620 19:00:37.513547 8192 consensus_queue.cc:237] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } }
I20260620 19:00:37.513969 8187 sys_catalog.cc:564] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:37.515250 8194 sys_catalog.cc:455] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df32be4bb685457d92f11fb1b0c7809a. Latest consensus state: current_term: 1 leader_uuid: "df32be4bb685457d92f11fb1b0c7809a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } } }
I20260620 19:00:37.515386 8194 sys_catalog.cc:458] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:37.515275 8193 sys_catalog.cc:455] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df32be4bb685457d92f11fb1b0c7809a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32be4bb685457d92f11fb1b0c7809a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39263 } } }
I20260620 19:00:37.515466 8193 sys_catalog.cc:458] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a [sys.catalog]: This master's current role is: LEADER
W20260620 19:00:37.518886 8207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:37.518973 8207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:37.519063 8202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:37.519942 8202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:37.525808 8202 catalog_manager.cc:1347] Generated new cluster ID: 0c87eec3d87c4939a10752e9449d6539
I20260620 19:00:37.525883 8202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:37.536870 8202 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:37.537796 8202 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:37.541819 8202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df32be4bb685457d92f11fb1b0c7809a: Generated new TSK 0
I20260620 19:00:37.542459 8202 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:00:37.563786 8217 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:00:37.563872 8217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:00:37.564474 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.549021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.2.208.190:39263)
0620 19:00:37.549170 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.549175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.549345 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:00:37.549488 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.549490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.549532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.549616 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:00:37.549873 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.550396 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.551616 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.551831 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.551927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.552249 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:00:37.564163 (+ 11914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:00:37.564372 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:00:37.564553 8190 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:00:37.564739 8216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:37.548955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39263 (local address 127.0.0.1:59390)
0620 19:00:37.549256 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:37.549271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:37.549381 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:37.549634 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:37.549637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:37.549645 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:00:37.549774 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.549780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.550513 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.550516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.551508 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.551521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.551963 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.551967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.552061 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.552090 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:00:37.564346 (+ 12256us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:00:37.564545 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.208.190:39263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260620 19:00:37.565470 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8120
2026-06-20T19:00:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:00:37.604718 2882 mini_oidc.cc:152] Starting JWKS server
I20260620 19:00:37.605580 2882 webserver.cc:460] Webserver started at https://127.0.0.1:46053/ using document root <none> and password file <none>
I20260620 19:00:37.605715 2882 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:00:37.605983 2882 webserver.cc:460] Webserver started at http://127.0.0.1:42337/ using document root <none> and password file <none>
2026-06-20T19:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:37Z Disabled control of system clock
I20260620 19:00:37.614472 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:33407
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:33407
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46053/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:00:37.720852 8223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:37.721103 8223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:37.721204 8223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:37.725075 8223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:37.725135 8223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:37.725152 8223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:37.725209 8223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:37.725245 8223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:00:37.725281 8223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:00:37.725330 8223 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46053/jwks.json
I20260620 19:00:37.730000 8223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:33407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46053/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:37.730997 8223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:37.732336 8223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:37.740648 8228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:37.740792 8223 server_base.cc:1034] running on GCE node
W20260620 19:00:37.740691 8229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:37.740659 8231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:37.741465 8223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:37.742509 8223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:37.743707 8223 hybrid_clock.cc:648] HybridClock initialized: now 1781982037743691 us; error 32 us; skew 500 ppm
I20260620 19:00:37.745383 8223 webserver.cc:460] Webserver started at http://127.2.208.190:34337/ using document root <none> and password file <none>
I20260620 19:00:37.745949 8223 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:37.746038 8223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:37.746265 8223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:37.748370 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "2e6a2f890b69487d9c4b0f82a131c59a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "149fdbacb4120483e5dae1d388a7f470"
server_key_iv: "c03afdc336c62480f4cdcb62889552b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.748915 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "2e6a2f890b69487d9c4b0f82a131c59a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "149fdbacb4120483e5dae1d388a7f470"
server_key_iv: "c03afdc336c62480f4cdcb62889552b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.752406 8223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:37.754720 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:37.755885 8223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:37.756027 8223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2e6a2f890b69487d9c4b0f82a131c59a"
format_stamp: "Formatted at 2026-06-20 19:00:37 on dist-test-slave-tsm4"
server_key: "149fdbacb4120483e5dae1d388a7f470"
server_key_iv: "c03afdc336c62480f4cdcb62889552b9"
server_key_version: "encryptionkey@0"
I20260620 19:00:37.756131 8223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:37.774410 8223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:37.775113 8223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:37.775319 8223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:37.782294 8223 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:33407
I20260620 19:00:37.782320 8289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:33407 every 8 connection(s)
I20260620 19:00:37.783257 8223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:37.785987 8290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:37.789108 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8223
I20260620 19:00:37.789285 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:37.789537 2882 external_mini_cluster.cc:1428] Setting key 3eb5f1869e382ea9cff0cbf9a28dde5a
I20260620 19:00:37.791901 8290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a: Bootstrap starting.
I20260620 19:00:37.794555 8290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:37.795420 8290 log.cc:824] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:37.797477 8290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a: No bootstrap required, opened a new log
I20260620 19:00:37.799811 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.790841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.2.208.190:33407)
0620 19:00:37.791563 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.791582 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.791623 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:00:37.792418 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.792440 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.792950 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.793312 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:37.793760 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.795215 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.796323 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.796985 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.797135 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.797880 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:37.797886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:37.797901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:37.798351 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:37.798415 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:37.798439 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:37.798532 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:37.798622 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:00:37.798877 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260620 19:00:37.801764 8290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } }
I20260620 19:00:37.801996 8290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:37.802054 8290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6a2f890b69487d9c4b0f82a131c59a, State: Initialized, Role: FOLLOWER
I20260620 19:00:37.802546 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:00:37.802508 8290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } }
I20260620 19:00:37.802654 8290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:37.802712 8290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:37.802798 8290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:37.803822 8290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } }
I20260620 19:00:37.804152 8290 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6a2f890b69487d9c4b0f82a131c59a; no voters:
I20260620 19:00:37.804450 8290 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:37.804605 8295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:37.804863 8295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [term 1 LEADER]: Becoming Leader. State: Replica: 2e6a2f890b69487d9c4b0f82a131c59a, State: Running, Role: LEADER
I20260620 19:00:37.805171 8295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } }
I20260620 19:00:37.805598 8290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:37.806761 8296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } } }
I20260620 19:00:37.806879 8296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:37.807186 8297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6a2f890b69487d9c4b0f82a131c59a. Latest consensus state: current_term: 1 leader_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6a2f890b69487d9c4b0f82a131c59a" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33407 } } }
I20260620 19:00:37.807313 8297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:37.807271 8304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:37.810151 8304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:37.814769 8304 catalog_manager.cc:1347] Generated new cluster ID: ea5cb86ed7dd473ebf988b23f02565c3
I20260620 19:00:37.814831 8304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:37.827418 8304 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:37.828595 8304 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:37.840103 8304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6a2f890b69487d9c4b0f82a131c59a: Generated new TSK 0
I20260620 19:00:37.840803 8304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:00:37.855834 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:37.851017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.2.208.190:33407)
0620 19:00:37.851144 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:37.851151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:37.851307 (+ 156us) server_negotiation.cc:407] Connection header received
0620 19:00:37.851467 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:37.851469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:37.851513 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:37.851596 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:00:37.851836 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.852338 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.853549 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.853795 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.853873 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.854205 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:00:37.855439 (+ 1234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:00:37.855689 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48492: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 19:00:37.855901 8319 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:37.850926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:33407 (local address 127.0.0.1:48492)
0620 19:00:37.851210 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:37.851224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:37.851338 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:37.851621 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:37.851624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:37.851632 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:00:37.851739 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.851745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.852438 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.852442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.853419 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:37.853428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:37.853902 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:37.853906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:37.853996 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:37.854033 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:00:37.855592 (+ 1559us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:00:37.855726 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.208.190:33407: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260620 19:00:37.856225 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8223
E20260620 19:00:37.868461 8320 webserver.cc:533] Webserver: sslize failed:
E20260620 19:00:37.868556 8320 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:00:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:37Z Disabled control of system clock
I20260620 19:00:37.904038 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44687
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44687
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/testchainca.cert with env {}
W20260620 19:00:38.011927 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:38.012178 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:38.012217 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:38.016317 8324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:38.016371 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:38.016386 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:38.016419 8324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:38.016448 8324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:00:38.016472 8324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:00:38.016503 8324 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:00:38.021719 8324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:38.022729 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:38.023732 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:38.032184 8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:38.032169 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:38.032228 8324 server_base.cc:1034] running on GCE node
W20260620 19:00:38.032174 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:38.032920 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:38.034024 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:38.035203 8324 hybrid_clock.cc:648] HybridClock initialized: now 1781982038035182 us; error 34 us; skew 500 ppm
I20260620 19:00:38.037009 8324 webserver.cc:460] Webserver started at http://127.2.208.190:41783/ using document root <none> and password file <none>
I20260620 19:00:38.037569 8324 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:38.037657 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:38.037878 8324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:38.039919 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "7bd4118eddca408ab2806347b0ef14f6"
format_stamp: "Formatted at 2026-06-20 19:00:38 on dist-test-slave-tsm4"
server_key: "cf1d663ba16512a4456f4bc6f9a3703e"
server_key_iv: "5c565f23874b7130768a39fd96ff7276"
server_key_version: "encryptionkey@0"
I20260620 19:00:38.040496 8324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "7bd4118eddca408ab2806347b0ef14f6"
format_stamp: "Formatted at 2026-06-20 19:00:38 on dist-test-slave-tsm4"
server_key: "cf1d663ba16512a4456f4bc6f9a3703e"
server_key_iv: "5c565f23874b7130768a39fd96ff7276"
server_key_version: "encryptionkey@0"
I20260620 19:00:38.044010 8324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:38.046321 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:38.047580 8324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:38.047710 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "7bd4118eddca408ab2806347b0ef14f6"
format_stamp: "Formatted at 2026-06-20 19:00:38 on dist-test-slave-tsm4"
server_key: "cf1d663ba16512a4456f4bc6f9a3703e"
server_key_iv: "5c565f23874b7130768a39fd96ff7276"
server_key_version: "encryptionkey@0"
I20260620 19:00:38.047830 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:38.068595 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:38.069319 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:38.069520 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:38.076231 8324 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44687
I20260620 19:00:38.076239 8390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44687 every 8 connection(s)
I20260620 19:00:38.077168 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:38.079687 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8324
I20260620 19:00:38.079821 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:38.080128 2882 external_mini_cluster.cc:1428] Setting key e5374c118b4f388e6f4561ecd3895a14
I20260620 19:00:38.080502 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:38.087541 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6: Bootstrap starting.
I20260620 19:00:38.090292 8391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:38.090346 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:38.081444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.2.208.190:44687)
0620 19:00:38.082100 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:38.082130 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:38.082173 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:00:38.082898 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:38.082925 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:38.083448 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:38.083809 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:38.084246 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.085687 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.086530 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.087200 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.087393 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:38.088324 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:38.088330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:38.088345 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:00:38.088850 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:38.088921 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:00:38.088951 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:38.089047 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:38.089178 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:00:38.089429 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260620 19:00:38.091048 8391 log.cc:824] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:38.092975 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:00:38.093008 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6: No bootstrap required, opened a new log
I20260620 19:00:38.096740 8391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } }
I20260620 19:00:38.096999 8391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:38.097075 8391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd4118eddca408ab2806347b0ef14f6, State: Initialized, Role: FOLLOWER
I20260620 19:00:38.097538 8391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } }
I20260620 19:00:38.097692 8391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:38.097779 8391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:38.097878 8391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:38.098796 8391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } }
I20260620 19:00:38.099133 8391 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd4118eddca408ab2806347b0ef14f6; no voters:
I20260620 19:00:38.099493 8391 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:38.099562 8396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:38.099799 8396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd4118eddca408ab2806347b0ef14f6, State: Running, Role: LEADER
I20260620 19:00:38.100143 8396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } }
I20260620 19:00:38.100833 8391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:38.101890 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd4118eddca408ab2806347b0ef14f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } } }
I20260620 19:00:38.102031 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:38.102327 8397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd4118eddca408ab2806347b0ef14f6. Latest consensus state: current_term: 1 leader_uuid: "7bd4118eddca408ab2806347b0ef14f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd4118eddca408ab2806347b0ef14f6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44687 } } }
I20260620 19:00:38.102398 8405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:38.102419 8397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:38.105214 8405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:38.110620 8405 catalog_manager.cc:1347] Generated new cluster ID: ebd8d326364c472dbf7f990c8141a566
I20260620 19:00:38.110711 8405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:38.135013 8405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:38.136360 8405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:38.146312 8405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd4118eddca408ab2806347b0ef14f6: Generated new TSK 0
I20260620 19:00:38.147181 8405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:00:38.204826 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:38.158401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.2.208.190:44687)
0620 19:00:38.158535 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:38.158540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:38.158677 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:00:38.158833 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:38.158836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:38.158892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:38.159037 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:00:38.159253 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.160106 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.160940 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.161278 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.161391 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:38.161643 (+ 252us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:00:38.204303 (+ 42660us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:00:38.204665 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 19:00:38.204912 8394 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 19:00:38.204890 8420 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:38.158294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44687 (local address 127.0.0.1:53350)
0620 19:00:38.158594 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:38.158606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:38.158698 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:38.159041 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:38.159043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:38.159049 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:00:38.159143 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:38.159148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.160211 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.160213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:38.160828 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:38.160835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:38.161387 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:38.161389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:38.161454 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:38.161474 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:00:38.204555 (+ 43081us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:00:38.204746 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.208.190:44687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260620 19:00:38.205343 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8324
2026-06-20T19:00:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:38 dist-test-slave-tsm4 krb5kdc[8432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:38 dist-test-slave-tsm4 krb5kdc[8432](info): set up 2 sockets
Jun 20 19:00:38 dist-test-slave-tsm4 krb5kdc[8432](info): commencing operation
krb5kdc: starting...
W20260620 19:00:39.483747 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:39 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:39.600415 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38633
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:39.711978 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:39.712262 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:39.712340 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:39.716352 8445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:39.716439 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:39.716475 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:39.716529 8445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:39.716564 8445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:39.721865 8445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:39.722967 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:39.724006 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:39.732115 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:39.732154 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:39.732116 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:39.732254 8445 server_base.cc:1034] running on GCE node
I20260620 19:00:39.732937 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:39.733989 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:39.735148 8445 hybrid_clock.cc:648] HybridClock initialized: now 1781982039735118 us; error 42 us; skew 500 ppm
Jun 20 19:00:39 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982039, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:39.737711 8445 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:39.738547 8445 webserver.cc:460] Webserver started at http://127.2.208.190:39497/ using document root <none> and password file <none>
I20260620 19:00:39.739074 8445 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:39.739161 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:39.739563 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:39.741524 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1"
format_stamp: "Formatted at 2026-06-20 19:00:39 on dist-test-slave-tsm4"
server_key: "f9faeb1eaab09cf00d46f3a77be67fbd"
server_key_iv: "3c6f59973af1504299f106b4925ff996"
server_key_version: "encryptionkey@0"
I20260620 19:00:39.742043 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1"
format_stamp: "Formatted at 2026-06-20 19:00:39 on dist-test-slave-tsm4"
server_key: "f9faeb1eaab09cf00d46f3a77be67fbd"
server_key_iv: "3c6f59973af1504299f106b4925ff996"
server_key_version: "encryptionkey@0"
I20260620 19:00:39.745517 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:39.747817 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:39.749032 8445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:39.749186 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1"
format_stamp: "Formatted at 2026-06-20 19:00:39 on dist-test-slave-tsm4"
server_key: "f9faeb1eaab09cf00d46f3a77be67fbd"
server_key_iv: "3c6f59973af1504299f106b4925ff996"
server_key_version: "encryptionkey@0"
I20260620 19:00:39.749305 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:39.792115 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:39.795025 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:39.795236 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:39.802162 8445 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:38633
I20260620 19:00:39.802186 8512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:38633 every 8 connection(s)
I20260620 19:00:39.803103 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:39.805962 8513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:39.806345 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8445
I20260620 19:00:39.806448 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:39.806718 2882 external_mini_cluster.cc:1428] Setting key d3d0c134809ab6da276cd98d51cc5597
I20260620 19:00:39.812892 8513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1: Bootstrap starting.
I20260620 19:00:39.814998 8513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1: Neither blocks nor log segments found. Creating new log.
Jun 20 19:00:39 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:39.815913 8513 log.cc:824] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:39.817934 8513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1: No bootstrap required, opened a new log
I20260620 19:00:39.820595 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:39.808065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.2.208.190:38633)
0620 19:00:39.808457 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:39.808475 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:39.808524 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:00:39.809274 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:39.809297 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:39.809710 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:39.810075 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:39.810737 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:39.812221 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:39.813021 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:39.813480 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:39.813648 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:39.816252 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:39.816257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:39.816289 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:39.816321 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:39.818001 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:39.818469 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:39.818475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:39.818480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:39.818590 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:39.818859 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:39.818863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:39.818865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:39.819238 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:39.819448 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:39.819672 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 19:00:39.819939 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260620 19:00:39.821401 8513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } }
I20260620 19:00:39.821622 8513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:39.821704 8513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c46166e72b140ff8a5b1b47f0c1e6d1, State: Initialized, Role: FOLLOWER
I20260620 19:00:39.822155 8513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } }
I20260620 19:00:39.822305 8513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:39.822391 8513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:39.822489 8513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:39.823290 8513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } }
I20260620 19:00:39.823649 8513 leader_election.cc:302] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c46166e72b140ff8a5b1b47f0c1e6d1; no voters:
I20260620 19:00:39.823958 8513 leader_election.cc:288] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:39.824064 8518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:39.824290 8518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9c46166e72b140ff8a5b1b47f0c1e6d1, State: Running, Role: LEADER
I20260620 19:00:39.824607 8518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } }
I20260620 19:00:39.825117 8513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:39.826233 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c46166e72b140ff8a5b1b47f0c1e6d1. Latest consensus state: current_term: 1 leader_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } } }
I20260620 19:00:39.826370 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:39.826622 8519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c46166e72b140ff8a5b1b47f0c1e6d1" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38633 } } }
I20260620 19:00:39.826713 8519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:39.827399 8527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:39.830183 8527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:39.835170 8527 catalog_manager.cc:1347] Generated new cluster ID: caf68f9dc4c84e9eb91f3e01f4d44a45
I20260620 19:00:39.835229 8527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:39.842754 8527 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:39.843988 8527 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:39.858426 8527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c46166e72b140ff8a5b1b47f0c1e6d1: Generated new TSK 0
I20260620 19:00:39.859243 8527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:39.887598 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:39.995777 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:39.996028 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:39.996068 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:39.999900 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:39.999950 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:40.000038 8539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:40.005139 8539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:40.006181 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:40.007112 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:40.016215 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:40.016223 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:40.016198 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:40.016530 8539 server_base.cc:1034] running on GCE node
I20260620 19:00:40.016913 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:40.017457 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:40.018602 8539 hybrid_clock.cc:648] HybridClock initialized: now 1781982040018569 us; error 46 us; skew 500 ppm
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.021147 8539 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:40.021953 8539 webserver.cc:460] Webserver started at http://127.2.208.129:35451/ using document root <none> and password file <none>
I20260620 19:00:40.022467 8539 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:40.022519 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:40.022671 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:40.024570 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "2ab81138eba34111b0a1c793c96d15eb"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "84d6a31565e5b0cc990937e0834ac733"
server_key_iv: "3b6b8e72c6dcbe979ed0ac97bf4a6b61"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.025128 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "2ab81138eba34111b0a1c793c96d15eb"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "84d6a31565e5b0cc990937e0834ac733"
server_key_iv: "3b6b8e72c6dcbe979ed0ac97bf4a6b61"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.028769 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:40.030982 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.032183 8539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:40.032300 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "2ab81138eba34111b0a1c793c96d15eb"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "84d6a31565e5b0cc990937e0834ac733"
server_key_iv: "3b6b8e72c6dcbe979ed0ac97bf4a6b61"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.032401 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:40.044054 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:40.046785 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:40.046989 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:40.047581 8539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:40.048406 8539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:40.048452 8539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.048519 8539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:40.048549 8539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.057610 8539 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:37851
I20260620 19:00:40.057634 8667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:37851 every 8 connection(s)
I20260620 19:00:40.058447 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:40.063324 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8539
I20260620 19:00:40.063496 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:40.063843 2882 external_mini_cluster.cc:1428] Setting key aefc893f4fcf9ae6b3231dcaa960ed19
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:40.072835 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.060173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:44295 (local address 127.2.208.190:38633)
0620 19:00:40.060309 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.060313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.061188 (+ 875us) server_negotiation.cc:407] Connection header received
0620 19:00:40.062435 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.062438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.062484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.062563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.063790 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.064474 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.065695 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.065939 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.066062 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.068860 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.068866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.068873 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.068906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.070722 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.071186 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.071191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.071193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.071285 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.071674 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.071679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.071682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.071924 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.072039 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.072499 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.072633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:40.073534 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.060541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38633 (local address 127.2.208.129:44295)
0620 19:00:40.060966 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:40.061005 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:40.062230 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:40.062717 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:40.062724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:40.063049 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:40.063633 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.063648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.064612 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.064614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.065573 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.065582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.066095 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.066099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.066281 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.066867 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:40.066882 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:40.068675 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:40.070835 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.070839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.070849 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.071085 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.071437 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.071441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.071445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.071574 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.072032 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:40.072036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:40.072248 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 19:00:40.072392 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 19:00:40.072626 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260620 19:00:40.074658 8668 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38633
I20260620 19:00:40.074913 8668 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:40.075430 8668 heartbeater.cc:511] Master 127.2.208.190:38633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:00:40.076886 8476 ts_manager.cc:194] Registered new tserver with Master: 2ab81138eba34111b0a1c793c96d15eb (127.2.208.129:37851)
I20260620 19:00:40.077968 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:44295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:40.090039 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:40.197161 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:40.197402 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:40.197441 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:40.201360 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:40.201411 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:40.201464 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:40.206543 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:40.207542 8673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:40.208509 8673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:40.217787 8678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:40.217849 8673 server_base.cc:1034] running on GCE node
W20260620 19:00:40.217851 8679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:40.217957 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:40.218441 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:40.218982 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:40.220161 8673 hybrid_clock.cc:648] HybridClock initialized: now 1781982040220135 us; error 39 us; skew 500 ppm
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.226151 8673 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:40.227072 8673 webserver.cc:460] Webserver started at http://127.2.208.130:35871/ using document root <none> and password file <none>
I20260620 19:00:40.227670 8673 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:40.227754 8673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:40.227983 8673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:40.230016 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "b10533aafdde4ec096623ed1494c7c54"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "4892b2163f8ae5aa51ed5f91cef9b169"
server_key_iv: "8d988517094b1dcb676245e8b784110b"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.230571 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "b10533aafdde4ec096623ed1494c7c54"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "4892b2163f8ae5aa51ed5f91cef9b169"
server_key_iv: "8d988517094b1dcb676245e8b784110b"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.234145 8673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 19:00:40.236518 8688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.237656 8673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:40.237792 8673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "b10533aafdde4ec096623ed1494c7c54"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "4892b2163f8ae5aa51ed5f91cef9b169"
server_key_iv: "8d988517094b1dcb676245e8b784110b"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.237912 8673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:40.263854 8673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:40.266579 8673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:40.266774 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:40.267333 8673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:40.268201 8673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:40.268268 8673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.268340 8673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:40.268381 8673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.277320 8673 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:46737
I20260620 19:00:40.277339 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:46737 every 8 connection(s)
I20260620 19:00:40.278179 8673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:40.286289 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8673
I20260620 19:00:40.286432 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:40.286691 2882 external_mini_cluster.cc:1428] Setting key 62b8983c15a0cf807bc775bbe4d39b43
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:40.291994 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.279896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:53393 (local address 127.2.208.190:38633)
0620 19:00:40.280053 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.280058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.280883 (+ 825us) server_negotiation.cc:407] Connection header received
0620 19:00:40.282244 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.282247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.282291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.282362 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.283493 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.284012 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.285094 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.285323 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.285403 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.288039 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.288043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.288045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.288075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.289736 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.290384 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.290386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.290387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.290460 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.290872 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.290875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.290876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.291044 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.291114 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.291783 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.291882 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:00:40.293207 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.280308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38633 (local address 127.2.208.130:53393)
0620 19:00:40.280726 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:40.280765 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:40.282063 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:40.282509 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:40.282516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:40.282850 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:40.283351 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.283383 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.284114 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.284116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.284988 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.284996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.285447 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.285449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.285578 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.286227 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:40.286242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:40.287852 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:40.289860 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.289865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.289876 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.290284 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.290549 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.290553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.290556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.290771 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.291145 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:40.291152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:40.291511 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 19:00:40.291736 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 19:00:40.292076 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260620 19:00:40.294386 8802 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38633
I20260620 19:00:40.294615 8802 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:40.295100 8802 heartbeater.cc:511] Master 127.2.208.190:38633 requested a full tablet report, sending...
I20260620 19:00:40.296160 8476 ts_manager.cc:194] Registered new tserver with Master: b10533aafdde4ec096623ed1494c7c54 (127.2.208.130:46737)
I20260620 19:00:40.296712 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:53393
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:40.314157 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:40.435998 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:40.436254 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:40.436342 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:40.440483 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:40.440534 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:40.440650 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:40.445876 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:40.446874 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:40.447889 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:40.457216 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:40.457237 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:40.457273 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:40.457228 8807 server_base.cc:1034] running on GCE node
I20260620 19:00:40.457847 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:40.458392 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:40.459540 8807 hybrid_clock.cc:648] HybridClock initialized: now 1781982040459525 us; error 28 us; skew 500 ppm
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.462167 8807 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:40.462983 8807 webserver.cc:460] Webserver started at http://127.2.208.131:46303/ using document root <none> and password file <none>
I20260620 19:00:40.463584 8807 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:40.463670 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:40.463891 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:40.465909 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "61228b469c89449ba455e21439cb8a85"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "29ee6d521a792dacff1c162df5943dcc"
server_key_iv: "5df3896e87df9ce7b6e6df1d300dd715"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.466480 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "61228b469c89449ba455e21439cb8a85"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "29ee6d521a792dacff1c162df5943dcc"
server_key_iv: "5df3896e87df9ce7b6e6df1d300dd715"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.470124 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:40.472458 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.473620 8807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:40.473764 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "61228b469c89449ba455e21439cb8a85"
format_stamp: "Formatted at 2026-06-20 19:00:40 on dist-test-slave-tsm4"
server_key: "29ee6d521a792dacff1c162df5943dcc"
server_key_iv: "5df3896e87df9ce7b6e6df1d300dd715"
server_key_version: "encryptionkey@0"
I20260620 19:00:40.473873 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:40.509727 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:40.513046 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:40.513268 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:40.513827 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:40.514638 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:40.514714 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.514796 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:40.514868 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:40.523283 8807 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:33053
I20260620 19:00:40.523306 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:33053 every 8 connection(s)
I20260620 19:00:40.524125 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:40.530392 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8807
I20260620 19:00:40.530527 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:40.530767 2882 external_mini_cluster.cc:1428] Setting key 03c4477830530786d5363c07dfbe17e6
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:40.537988 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.525978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:44403 (local address 127.2.208.190:38633)
0620 19:00:40.526136 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.526141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.527047 (+ 906us) server_negotiation.cc:407] Connection header received
0620 19:00:40.528388 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.528391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.528436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.528508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.529786 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.530326 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.531763 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.532058 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.532169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.534576 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.534579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.534581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.534603 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.536081 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.536567 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.536570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.536572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.536650 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.536993 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.536998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.537000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.537174 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.537278 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.537716 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.537828 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:00:40.538748 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.526416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38633 (local address 127.2.208.131:44403)
0620 19:00:40.526902 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:40.526943 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:40.528202 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:40.528692 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:40.528699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:40.529174 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:40.529640 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.529654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.530453 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.530458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.531635 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:40.531647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.532203 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.532207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:40.532383 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.533010 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:40.533026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:40.534397 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:40.536197 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.536202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.536215 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.536459 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.536740 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:40.536744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:40.536746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:40.536900 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:40.537276 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:40.537283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:40.537501 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 19:00:40.537678 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 19:00:40.537967 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260620 19:00:40.539951 8936 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38633
I20260620 19:00:40.540191 8936 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:40.540634 8936 heartbeater.cc:511] Master 127.2.208.190:38633 requested a full tablet report, sending...
I20260620 19:00:40.541671 8476 ts_manager.cc:194] Registered new tserver with Master: 61228b469c89449ba455e21439cb8a85 (127.2.208.131:33053)
I20260620 19:00:40.542373 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:44403
I20260620 19:00:40.544449 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/test-admin.keytab.
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.559438 2882 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:00:40.562155 8943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:40.560713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.2.208.129:37851)
0620 19:00:40.561023 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.561038 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.561088 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:00:40.561135 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.561140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.561314 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.561468 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.561945 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48570: BlockingRecv error: recv got EOF from 127.0.0.1:48570 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260620 19:00:40.562242 2882 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.563236 2882 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.564188 2882 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:40.573767 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.565048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.2.208.129:37851)
0620 19:00:40.565173 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.565177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.565279 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:00:40.565421 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.565430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.565469 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.565553 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.566059 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.566870 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.567779 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.568157 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.568281 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.570527 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.570535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.570547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.570568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.572031 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.572454 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.572467 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.572468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.572540 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.572809 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.572814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.572815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.573040 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.573240 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.573477 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.573629 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 19:00:40.577560 8943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:40.576489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.2.208.129:37851)
0620 19:00:40.576625 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.576629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.576727 (+ 98us) server_negotiation.cc:407] Connection header received
0620 19:00:40.576863 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.576866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.576904 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.577004 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.577441 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48586: BlockingRecv error: recv got EOF from 127.0.0.1:48586 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 19:00:40.577684 2882 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.578613 2882 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.579466 2882 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:40.586861 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.580218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.2.208.129:37851)
0620 19:00:40.580336 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.580339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.580366 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:00:40.580466 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.580467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.580497 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.580556 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.580957 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.581397 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.582085 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.582309 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.582381 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.584371 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.584374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.584375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.584395 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.585548 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.585899 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.585902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.585903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.585968 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.586181 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.586183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.586184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.586331 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.586421 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.586585 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.586708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:00:40.589335 8943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:40.588394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.2.208.129:37851)
0620 19:00:40.588505 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.588508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.588610 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:00:40.588754 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.588757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.588797 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.588874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.589250 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48594: BlockingRecv error: recv got EOF from 127.0.0.1:48594 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 19:00:40.589469 2882 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982008625501-2882-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.590401 2882 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:40.591262 2882 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:00:40.599404 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:40.592063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.2.208.129:37851)
0620 19:00:40.592172 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:40.592176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:40.592294 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:00:40.592405 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:40.592408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:40.592450 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:40.592532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:40.593010 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.593514 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.594270 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:40.594473 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:40.594576 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:40.596649 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:40.596654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:40.596661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:40.596681 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:40.597806 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.598185 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.598188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.598190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.598269 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:40.598601 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:40.598605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:40.598607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:40.598760 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:40.598888 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:40.599115 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 19:00:40.599230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:00:40.600463 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8539
I20260620 19:00:40.605890 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8673
I20260620 19:00:40.611238 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8807
I20260620 19:00:40.616432 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8445
2026-06-20T19:00:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2154 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8956](info): set up 2 sockets
Jun 20 19:00:40 dist-test-slave-tsm4 krb5kdc[8956](info): commencing operation
krb5kdc: starting...
W20260620 19:00:41.635784 2882 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:41 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab.
Entry for principal oryx/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab.
I20260620 19:00:41.751799 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39415
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab
--principal=oryx/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:41.860286 8969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:41.860541 8969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:41.860582 8969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:41.864513 8969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:41.864563 8969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:41.864583 8969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:41.864598 8969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:41.864614 8969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:41.869791 8969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab
--principal=oryx/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:41.870733 8969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:41.871690 8969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:41.879709 8974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:41.879696 8975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:41.879696 8977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:41.879756 8969 server_base.cc:1034] running on GCE node
I20260620 19:00:41.880501 8969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:41.881543 8969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:41.882730 8969 hybrid_clock.cc:648] HybridClock initialized: now 1781982041882717 us; error 26 us; skew 500 ppm
Jun 20 19:00:41 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:41.885340 8969 init.cc:377] Logged in from keytab as oryx/127.2.208.190@KRBTEST.COM (short username oryx)
I20260620 19:00:41.886103 8969 webserver.cc:460] Webserver started at http://127.2.208.190:35683/ using document root <none> and password file <none>
I20260620 19:00:41.886585 8969 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:41.886638 8969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:41.886801 8969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:41.888728 8969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "e8c6e1cbfd964be289e7b98e721c8c38"
format_stamp: "Formatted at 2026-06-20 19:00:41 on dist-test-slave-tsm4"
server_key: "9d09c8e93431c99858ed42276326d5ab"
server_key_iv: "06b3e64671684f5bbc2fe52c2c6fe44d"
server_key_version: "encryptionkey@0"
I20260620 19:00:41.889253 8969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "e8c6e1cbfd964be289e7b98e721c8c38"
format_stamp: "Formatted at 2026-06-20 19:00:41 on dist-test-slave-tsm4"
server_key: "9d09c8e93431c99858ed42276326d5ab"
server_key_iv: "06b3e64671684f5bbc2fe52c2c6fe44d"
server_key_version: "encryptionkey@0"
I20260620 19:00:41.892706 8969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:41.894982 8984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:41.896188 8969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:00:41.896307 8969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "e8c6e1cbfd964be289e7b98e721c8c38"
format_stamp: "Formatted at 2026-06-20 19:00:41 on dist-test-slave-tsm4"
server_key: "9d09c8e93431c99858ed42276326d5ab"
server_key_iv: "06b3e64671684f5bbc2fe52c2c6fe44d"
server_key_version: "encryptionkey@0"
I20260620 19:00:41.896405 8969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:41.908452 8969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:41.911229 8969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:41.911449 8969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:41.918265 8969 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39415
I20260620 19:00:41.918535 9036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39415 every 8 connection(s)
I20260620 19:00:41.919178 8969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:41.921865 9037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:41.927311 9037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Bootstrap starting.
I20260620 19:00:41.927858 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 8969
I20260620 19:00:41.927989 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:41.928268 2882 external_mini_cluster.cc:1428] Setting key b723e2c31e1be3b272c7680d490cff81
I20260620 19:00:41.929991 9037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:41.930720 9037 log.cc:824] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:41.932861 9037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: No bootstrap required, opened a new log
Jun 20 19:00:41 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:41.937549 9037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } }
I20260620 19:00:41.937794 9037 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:41.937881 9037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c6e1cbfd964be289e7b98e721c8c38, State: Initialized, Role: FOLLOWER
I20260620 19:00:41.938393 9037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } }
I20260620 19:00:41.938565 9037 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:41.938644 9037 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:41.938750 9037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:41.939968 9037 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } }
I20260620 19:00:41.940330 9037 leader_election.cc:302] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c6e1cbfd964be289e7b98e721c8c38; no voters:
I20260620 19:00:41.940693 9037 leader_election.cc:288] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:41.940788 9042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:41.940994 9042 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 LEADER]: Becoming Leader. State: Replica: e8c6e1cbfd964be289e7b98e721c8c38, State: Running, Role: LEADER
I20260620 19:00:41.941366 9042 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } }
I20260620 19:00:41.942020 9037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:41.943053 9043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c6e1cbfd964be289e7b98e721c8c38. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } }
I20260620 19:00:41.943159 9043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:41.943542 9044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } }
I20260620 19:00:41.943600 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:41.929512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.2.208.190:39415)
0620 19:00:41.929924 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:41.929940 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:41.929975 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:00:41.930627 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:41.930644 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:41.931020 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:41.931420 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:41.932295 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:41.933277 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:41.934414 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:41.934848 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:41.934957 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:41.937877 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:41.937882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:41.937895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:41.937920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:41.940020 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:41.940482 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:41.940488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:41.940493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:41.940649 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:41.940939 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:41.940942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:41.940944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:41.941409 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:41.941610 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:41.942224 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 19:00:41.942520 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260620 19:00:41.943701 9044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:41.943953 9051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:41.947105 9051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:41.952620 9051 catalog_manager.cc:1347] Generated new cluster ID: 6b2e7e34ec74411887986c4e09133417
I20260620 19:00:41.952682 9051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:41.961259 9051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:41.962096 9051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:41.965837 9051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Generated new TSK 0
I20260620 19:00:41.966454 9051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
I20260620 19:00:41.997766 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:42.105876 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:42.106112 9063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:42.106186 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:42.110152 9063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:42.110206 9063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:42.110296 9063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:42.115584 9063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:42.116585 9063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:42.117594 9063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:42.126883 9068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.126895 9069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.126895 9071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.127265 9063 server_base.cc:1034] running on GCE node
I20260620 19:00:42.127692 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:42.128281 9063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:42.129463 9063 hybrid_clock.cc:648] HybridClock initialized: now 1781982042129454 us; error 30 us; skew 500 ppm
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:42.134711 9063 init.cc:377] Logged in from keytab as oryx/127.2.208.129@KRBTEST.COM (short username oryx)
I20260620 19:00:42.135581 9063 webserver.cc:460] Webserver started at http://127.2.208.129:40757/ using document root <none> and password file <none>
I20260620 19:00:42.136124 9063 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.136236 9063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.136461 9063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:42.138505 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "d662bdfa7e804fedace36555fe20d717"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "374ae8b2a4887b625f50e8898134d374"
server_key_iv: "1a14095941cfe430a0ec5ae186359f65"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.139046 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "d662bdfa7e804fedace36555fe20d717"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "374ae8b2a4887b625f50e8898134d374"
server_key_iv: "1a14095941cfe430a0ec5ae186359f65"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.142683 9063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:42.145156 9078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.146303 9063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:42.146430 9063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "d662bdfa7e804fedace36555fe20d717"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "374ae8b2a4887b625f50e8898134d374"
server_key_iv: "1a14095941cfe430a0ec5ae186359f65"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.146533 9063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:42.158874 9063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:42.161459 9063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:42.161650 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:42.162225 9063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:42.163017 9063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:42.163085 9063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.163153 9063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:42.163180 9063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.171509 9063 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:42729
I20260620 19:00:42.171530 9191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:42729 every 8 connection(s)
I20260620 19:00:42.172370 9063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:42.173769 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9063
I20260620 19:00:42.173889 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:42.174193 2882 external_mini_cluster.cc:1428] Setting key 1d60c2988ea25148757ac2a3ab1ef95e
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.130@KRBTEST.COM" created.
I20260620 19:00:42.187031 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.174174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55595 (local address 127.2.208.190:39415)
0620 19:00:42.174313 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.174318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.175289 (+ 971us) server_negotiation.cc:407] Connection header received
0620 19:00:42.177197 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.177199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.177241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.177309 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.178516 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.179080 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.180279 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.180530 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.180639 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.183454 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.183457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.183459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.183481 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.185005 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.185442 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.185445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.185446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.185517 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.185833 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.185835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.185836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.186057 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.186133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.186626 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.186774 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:42.187692 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.174663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:55595)
0620 19:00:42.175121 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.175170 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.176973 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.177518 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.177530 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.177902 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.178365 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.178378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.179223 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.179226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.180146 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.180154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.180672 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.180676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.180842 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.181437 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.181453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.183222 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.185104 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.185108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.185119 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.185355 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.185595 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.185597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.185598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.185724 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.186157 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.186165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.186391 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.186544 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.186793 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":136,"threads_started":1}
I20260620 19:00:42.188922 9192 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:42.189174 9192 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:42.189638 9192 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
I20260620 19:00:42.191236 9001 ts_manager.cc:194] Registered new tserver with Master: d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
I20260620 19:00:42.192904 9001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:55595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
I20260620 19:00:42.201838 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:42.311878 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:42.312129 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:42.312171 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:42.316118 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:42.316171 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:42.316227 9197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:42.321470 9197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:42.322423 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:42.323414 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:42.332742 9203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.332726 9202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.332736 9205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.333112 9197 server_base.cc:1034] running on GCE node
I20260620 19:00:42.333521 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:42.334120 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:42.335312 9197 hybrid_clock.cc:648] HybridClock initialized: now 1781982042335297 us; error 26 us; skew 500 ppm
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:42.338073 9197 init.cc:377] Logged in from keytab as oryx/127.2.208.130@KRBTEST.COM (short username oryx)
I20260620 19:00:42.338841 9197 webserver.cc:460] Webserver started at http://127.2.208.130:34985/ using document root <none> and password file <none>
I20260620 19:00:42.339335 9197 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.339457 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.339643 9197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:42.341627 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "42bc808a023544749fd7aaf831414970"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "2377382029f74267a61d6005f23f8a77"
server_key_iv: "39a4fab92c4f09a928078192374340fd"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.342131 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "42bc808a023544749fd7aaf831414970"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "2377382029f74267a61d6005f23f8a77"
server_key_iv: "39a4fab92c4f09a928078192374340fd"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.345850 9197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:42.348148 9212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.349354 9197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:42.349483 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "42bc808a023544749fd7aaf831414970"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "2377382029f74267a61d6005f23f8a77"
server_key_iv: "39a4fab92c4f09a928078192374340fd"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.349591 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:42.372097 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:42.374972 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:42.375174 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:42.375764 9197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:42.376592 9197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:42.376637 9197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.376703 9197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:42.376740 9197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.385301 9197 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:38639
I20260620 19:00:42.385442 9325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:38639 every 8 connection(s)
I20260620 19:00:42.386200 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:42.387741 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9197
I20260620 19:00:42.387863 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:42.388182 2882 external_mini_cluster.cc:1428] Setting key 095d120a03dd684d8c374a2fd815a05d
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.131@KRBTEST.COM" created.
I20260620 19:00:42.401000 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.387948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59623 (local address 127.2.208.190:39415)
0620 19:00:42.388109 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.388115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.389000 (+ 885us) server_negotiation.cc:407] Connection header received
0620 19:00:42.390364 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.390367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.390416 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.390518 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.392086 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.392604 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.394303 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.394534 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.394750 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.397270 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.397274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.397277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.397303 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.398762 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.399199 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.399202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.399203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.399278 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.399737 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.399741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.399742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.399985 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.400093 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.400644 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.400836 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:42.401734 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.388382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:59623)
0620 19:00:42.388852 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.388903 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.390136 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.390682 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.390688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.391197 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.391923 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.391946 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.392724 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.392730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.394180 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.394193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.394783 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.394787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.394951 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.395642 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.395658 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.397095 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.398858 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.398863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.398874 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.399114 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.399351 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.399353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.399354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.399631 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.400112 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.400116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.400371 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.400573 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.400896 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260620 19:00:42.402990 9326 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:42.403226 9326 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:42.403750 9326 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
I20260620 19:00:42.404753 9001 ts_manager.cc:194] Registered new tserver with Master: 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
I20260620 19:00:42.405336 9001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:59623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
I20260620 19:00:42.415705 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:42.524936 9331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:42.525223 9331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:42.525269 9331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:42.529409 9331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:42.529461 9331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:42.529521 9331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:42.534685 9331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:42.535688 9331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:42.536679 9331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:42.546144 9339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.546180 9331 server_base.cc:1034] running on GCE node
W20260620 19:00:42.546170 9337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.546177 9336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.546867 9331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:42.547586 9331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:42.548740 9331 hybrid_clock.cc:648] HybridClock initialized: now 1781982042548735 us; error 36 us; skew 500 ppm
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:42.552158 9331 init.cc:377] Logged in from keytab as oryx/127.2.208.131@KRBTEST.COM (short username oryx)
I20260620 19:00:42.553180 9331 webserver.cc:460] Webserver started at http://127.2.208.131:33647/ using document root <none> and password file <none>
I20260620 19:00:42.553788 9331 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.553877 9331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.554078 9331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:42.556131 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "e596d9f6904443e19f9dca54132ec024"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "b33968b298dfd0f22363bd89a2770f1e"
server_key_iv: "9b00a4ab40bfe8e656d32f1c9b12a144"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.556696 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "e596d9f6904443e19f9dca54132ec024"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "b33968b298dfd0f22363bd89a2770f1e"
server_key_iv: "9b00a4ab40bfe8e656d32f1c9b12a144"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.560328 9331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:42.562683 9346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.563934 9331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:42.564064 9331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e596d9f6904443e19f9dca54132ec024"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "b33968b298dfd0f22363bd89a2770f1e"
server_key_iv: "9b00a4ab40bfe8e656d32f1c9b12a144"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.564173 9331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:42.586634 9331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:42.589375 9331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:42.589571 9331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:42.590133 9331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:42.590929 9331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:42.590996 9331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.591063 9331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:42.591110 9331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.599318 9331 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:44091
I20260620 19:00:42.599351 9459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:44091 every 8 connection(s)
I20260620 19:00:42.600294 9331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:42.601464 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9331
I20260620 19:00:42.601545 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:42.601816 2882 external_mini_cluster.cc:1428] Setting key 99134298b2f5fad8094997a3885d2534
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:42.618743 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.602141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:60347 (local address 127.2.208.190:39415)
0620 19:00:42.602275 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.602286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.603300 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 19:00:42.604805 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.604808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.604861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.604938 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.606278 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.607211 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.609189 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.609652 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.609781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.614593 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.614597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.614601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.614628 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.616499 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.617030 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.617035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.617038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.617133 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.617464 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.617470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.617474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.617730 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.617832 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.618363 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.618505 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:42.619577 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.602581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:60347)
0620 19:00:42.603146 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.603203 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.604604 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.605102 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.605109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.605457 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.606096 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.606118 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.607395 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.607400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.608988 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.608998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.609980 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.609986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.610247 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.610954 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.610993 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.614426 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.616610 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.616614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.616625 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.616918 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.617222 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.617225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.617226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.617364 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.617828 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.617833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.618110 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.618272 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.618540 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":138,"threads_started":1}
I20260620 19:00:42.620711 9460 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:42.620946 9460 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:42.621431 9460 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
I20260620 19:00:42.622529 9001 ts_manager.cc:194] Registered new tserver with Master: e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091)
I20260620 19:00:42.623431 9001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:60347
I20260620 19:00:42.627431 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.208.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.189.keytab.
Entry for principal oryx/127.2.208.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.189.keytab.
I20260620 19:00:42.654039 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:39873
--webserver_interface=127.2.208.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.189.keytab
--principal=oryx/127.2.208.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39415,127.2.208.189:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:42.763197 9465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:42.763479 9465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:42.763547 9465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:42.767647 9465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:42.767699 9465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:42.767729 9465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:42.767745 9465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:42.767791 9465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:42.772977 9465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39415,127.2.208.189:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.189.keytab
--principal=oryx/127.2.208.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.208.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
W20260620 19:00:42.773178 9465 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.208.190:39415,127.2.208.189:39873'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:00:42.774020 9465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:42.775013 9465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:42.783015 9470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.783022 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.783118 9473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.783947 9465 server_base.cc:1034] running on GCE node
I20260620 19:00:42.784534 9465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:42.785626 9465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:42.786805 9465 hybrid_clock.cc:648] HybridClock initialized: now 1781982042786786 us; error 37 us; skew 500 ppm
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:42.789880 9465 init.cc:377] Logged in from keytab as oryx/127.2.208.189@KRBTEST.COM (short username oryx)
I20260620 19:00:42.791112 9465 webserver.cc:460] Webserver started at http://127.2.208.189:40257/ using document root <none> and password file <none>
I20260620 19:00:42.791873 9465 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.791981 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.792258 9465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:42.795235 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/instance:
uuid: "4e3933b420ab43a1841db9e0d591a974"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "05ce7729a5514ecd1d1d16a301cda82f"
server_key_iv: "35f0c67415efd61e6e77efd868fbff4a"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.796005 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal/instance:
uuid: "4e3933b420ab43a1841db9e0d591a974"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "05ce7729a5514ecd1d1d16a301cda82f"
server_key_iv: "35f0c67415efd61e6e77efd868fbff4a"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.801478 9465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:42.804730 9480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.806342 9465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:00:42.806504 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "4e3933b420ab43a1841db9e0d591a974"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "05ce7729a5514ecd1d1d16a301cda82f"
server_key_iv: "35f0c67415efd61e6e77efd868fbff4a"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.806660 9465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:42.827806 9465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:42.831775 9465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:42.831959 9465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:42.838909 9465 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.189:39873
I20260620 19:00:42.839185 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.189:39873 every 8 connection(s)
I20260620 19:00:42.839943 9465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
I20260620 19:00:42.842676 9533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:42.846387 9533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:00:42.850035 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9465
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.189@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:42.860047 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.848228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.2.208.190:39415)
0620 19:00:42.848361 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.848365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.849126 (+ 761us) server_negotiation.cc:407] Connection header received
0620 19:00:42.850112 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.850117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.850178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.850265 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.851703 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.852224 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.853391 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.853595 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.853705 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.856452 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.856455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.856457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.856493 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.857906 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.858460 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.858473 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.858474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.858546 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.858873 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.858878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.858879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.859090 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.859199 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.859759 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.859875 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:42.861016 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.848534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37524)
0620 19:00:42.848946 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.848985 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.849854 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.850472 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.850481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.850906 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.851553 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.851568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.852370 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.852373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.853260 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.853268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.853727 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.853731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.853895 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.854484 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.854499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.856304 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.858027 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.858037 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.858047 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.858350 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.858635 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.858640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.858642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.858778 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.859202 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.859211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.859517 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.859699 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.859997 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
I20260620 19:00:42.861020 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.850908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.2.208.189:39873)
0620 19:00:42.851291 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.851300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.851319 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:00:42.851414 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.851420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.851580 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.851744 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.852431 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.853289 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.854059 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.854409 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.854519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.857451 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.857456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.857466 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.857487 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.859095 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.859611 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.859621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.859622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.859700 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.859987 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.859991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.859992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.860289 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.860464 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.860675 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.860781 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260620 19:00:42.862372 9533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.189@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:42.869807 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.862865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42270)
0620 19:00:42.863035 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.863049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.863121 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.863323 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.863325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.863505 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.863585 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.863590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.864247 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.864249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.864740 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.864746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.865140 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.865142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.865213 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.865578 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.865588 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.866705 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.868546 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.868549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.868550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.868791 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.869046 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.869048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.869049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.869129 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.869425 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.869426 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.869496 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.869571 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.869672 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 19:00:42.869838 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.862888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.2.208.189:39873)
0620 19:00:42.863096 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.863099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.863110 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:00:42.863198 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.863199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.863231 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.863290 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.863690 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.864146 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.864829 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.865042 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.865123 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.866837 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.866839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.866841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.866860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.868426 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.868887 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.868889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.868891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.868958 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.869202 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.869204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.869205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.869349 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.869420 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.869601 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.869723 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":142}
I20260620 19:00:42.873054 9533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Bootstrap starting.
I20260620 19:00:42.875352 9533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:42.876160 9533 log.cc:824] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:42.878010 9533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: No bootstrap required, opened a new log
I20260620 19:00:42.881032 9533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } }
I20260620 19:00:42.881265 9533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:42.881345 9533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3933b420ab43a1841db9e0d591a974, State: Initialized, Role: FOLLOWER
I20260620 19:00:42.881821 9533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } }
I20260620 19:00:42.882287 9539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } } }
I20260620 19:00:42.882395 9539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:42.883003 9533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:00:42.887802 9550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:42.887897 9550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:42.890065 9465 master_runner.cc:428] Detected that this master 4e3933b420ab43a1841db9e0d591a974 is joining an existing cluster
I20260620 19:00:42.890146 9465 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.208.189:39873
I20260620 19:00:42.900864 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.892277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.2.208.190:39415)
0620 19:00:42.892418 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.892423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.892852 (+ 429us) server_negotiation.cc:407] Connection header received
0620 19:00:42.892997 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.893000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.893046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.893139 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.893632 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.894165 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.894997 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.895261 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.895405 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.897244 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.897248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.897251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.897281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.899022 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.899522 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.899526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.899528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.899628 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.899940 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.899943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.899945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.900179 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.900286 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.900572 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.900699 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:00:42.901358 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.892240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37540)
0620 19:00:42.892763 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.892776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.892902 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.893185 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.893188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.893395 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.893511 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.893516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.894274 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.894279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.894878 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.894886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.895403 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.895407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.895495 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.896571 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.896587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.897098 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.899147 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.899151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.899152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.899420 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.899720 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.899723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.899725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.899828 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.900277 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.900280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.900372 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.901048 (+ 676us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.901183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"spinlock_wait_cycles":6016,"thread_start_us":116,"threads_started":1}
I20260620 19:00:42.903896 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.892329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.2.208.189:39873)
0620 19:00:42.892434 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.892438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.896264 (+ 3826us) server_negotiation.cc:407] Connection header received
0620 19:00:42.896353 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.896357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.896415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.896508 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.897051 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.897701 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.898676 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.898889 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.898995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.900310 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.900313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.900315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.900335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.902055 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.902709 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.902713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.902714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.902786 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.903041 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.903043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.903045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.903208 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.903272 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.903594 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.903720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:00:42.904345 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.892506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42272)
0620 19:00:42.896158 (+ 3652us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.896173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.896270 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.896543 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.896546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.896821 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.896932 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.896939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.897819 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.897824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.898506 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.898515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.898995 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.898998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.899100 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.899757 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.899768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.900179 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.902350 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.902353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.902355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.902581 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.902881 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.902884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.902886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.902957 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.903325 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.903328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.903434 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.904028 (+ 594us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.904155 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3575,"spinlock_wait_cycles":1920,"thread_start_us":97,"threads_started":1}
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.190@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:42.915539 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.906158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.2.208.189:39873)
0620 19:00:42.906252 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.906256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.906563 (+ 307us) server_negotiation.cc:407] Connection header received
0620 19:00:42.906738 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.906741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.906789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.906875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.907510 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.908262 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.909105 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.909431 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.909534 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.911967 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.911970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.911972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.911991 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.913719 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.914252 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.914255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.914257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.914336 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.914623 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.914627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.914629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.914802 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.914908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.915248 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.915353 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:00:42.915573 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.906116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42282)
0620 19:00:42.906440 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.906487 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.906614 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.906911 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.906915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.907253 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.907350 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.907403 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.908371 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.908373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.908986 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.908994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.909539 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.909541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.909616 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.910122 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.910133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.911823 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.913818 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.913828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.913835 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.914166 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.914436 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.914439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.914441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.914529 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.914910 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.914919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.915046 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.915200 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.915414 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260620 19:00:42.917371 9001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.208.189:39873: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4e3933b420ab43a1841db9e0d591a974"
member_type: NON_VOTER
last_known_addr {
host: "127.2.208.189"
port: 39873
}
attrs {
promote: true
}
}
dest_uuid: "e8c6e1cbfd964be289e7b98e721c8c38"
cas_config_opid_index: -1
I20260620 19:00:42.918038 9001 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } }
I20260620 19:00:42.920063 9561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } }
I20260620 19:00:42.920673 9507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:42.921622 9561 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.208.189:39873
I20260620 19:00:42.921803 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } }
I20260620 19:00:42.921895 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:42.922027 9507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:00:42.922976 9465 master.cc:537] Master@127.2.208.189:39873 shutting down...
I20260620 19:00:42.922969 9561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:00:42.925279 8987 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 19:00:42.928395 9465 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:42.928675 9465 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:42.928762 9465 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: stopping tablet replica
W20260620 19:00:42.929190 8987 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 -> Peer 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189:39873): Couldn't send request to peer 4e3933b420ab43a1841db9e0d591a974. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.2.208.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.188@KRBTEST.COM" created.
I20260620 19:00:42.943712 9465 master.cc:552] Master@127.2.208.189:39873 shutdown complete.
I20260620 19:00:42.943776 9465 master_runner.cc:305] Clearing existing system tablet
I20260620 19:00:42.944029 9465 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.944087 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.945405 9465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:42.946965 9566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.947247 9465 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:42.947394 9465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:42.947431 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "4e3933b420ab43a1841db9e0d591a974"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "05ce7729a5514ecd1d1d16a301cda82f"
server_key_iv: "35f0c67415efd61e6e77efd868fbff4a"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.950017 9465 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.188.keytab.
Entry for principal oryx/127.2.208.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.188.keytab.
I20260620 19:00:42.953365 9465 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:00:42.953462 9465 log.cc:1196] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 19:00:42.953730 9465 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Deleting consensus metadata
I20260620 19:00:42.953896 9465 master_runner.cc:315] Copying system tablet from 127.2.208.190:39415
I20260620 19:00:42.954027 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:33035
--webserver_interface=127.2.208.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.188.keytab
--principal=oryx/127.2.208.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:00:42.955051 9465 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: tablet copy: Beginning tablet copy session from remote peer at address 127.2.208.190:39415
I20260620 19:00:42.963617 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.955908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.2.208.190:39415)
0620 19:00:42.956063 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:42.956070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:42.956346 (+ 276us) server_negotiation.cc:407] Connection header received
0620 19:00:42.956480 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:42.956483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:42.956539 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:42.956631 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:42.957143 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.957962 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.958772 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.959112 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.959218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.960472 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:42.960476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:42.960479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:42.960506 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:42.962024 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.962418 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.962421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.962423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.962510 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:42.962763 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:42.962766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:42.962768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:42.962970 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:42.963065 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:42.963335 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 19:00:42.963471 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:42.963629 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:42.955780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37542)
0620 19:00:42.956218 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:42.956234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:42.956374 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:42.956671 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:42.956675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:42.956909 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:42.957019 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.957033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.958083 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.958085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.958662 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:42.958670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:42.959229 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:42.959232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:42.959305 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:42.960013 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:42.960025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:42.960332 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:42.962135 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.962137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.962139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.962328 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.962596 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:42.962598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:42.962599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:42.962676 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:42.963085 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:42.963087 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:42.963173 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 19:00:42.963264 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:00:42.963443 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":89,"threads_started":1}
I20260620 19:00:42.964280 9021 tablet_copy_service.cc:140] P e8c6e1cbfd964be289e7b98e721c8c38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4e3933b420ab43a1841db9e0d591a974 ({username='oryx', principal='oryx/127.2.208.189@KRBTEST.COM'} at 127.0.0.1:37542)
I20260620 19:00:42.964470 9021 tablet_copy_service.cc:161] P e8c6e1cbfd964be289e7b98e721c8c38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4e3933b420ab43a1841db9e0d591a974 at {username='oryx', principal='oryx/127.2.208.189@KRBTEST.COM'} at 127.0.0.1:37542: session id = 4e3933b420ab43a1841db9e0d591a974-00000000000000000000000000000000
I20260620 19:00:42.966284 9021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:00:42.968171 9465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:00:42.971079 9465 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: tablet copy: Starting download of 0 data blocks...
I20260620 19:00:42.971293 9465 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: tablet copy: Starting download of 1 WAL segments...
I20260620 19:00:42.973192 9465 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:00:42.974984 9021 tablet_copy_service.cc:342] P e8c6e1cbfd964be289e7b98e721c8c38: Request end of tablet copy session 4e3933b420ab43a1841db9e0d591a974-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.208.189@KRBTEST.COM'} at 127.0.0.1:37542
I20260620 19:00:42.975106 9021 tablet_copy_service.cc:433] P e8c6e1cbfd964be289e7b98e721c8c38: ending tablet copy session 4e3933b420ab43a1841db9e0d591a974-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4e3933b420ab43a1841db9e0d591a974
I20260620 19:00:42.982163 9465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:42.983826 9574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.984315 9575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:42.984594 9577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:42.985221 9465 server_base.cc:1034] running on GCE node
I20260620 19:00:42.985378 9465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:42.985664 9465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:42.986871 9465 hybrid_clock.cc:648] HybridClock initialized: now 1781982042986848 us; error 40 us; skew 500 ppm
Jun 20 19:00:42 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982042, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:42.988514 9465 init.cc:377] Logged in from keytab as oryx/127.2.208.189@KRBTEST.COM (short username oryx)
I20260620 19:00:42.988986 9465 webserver.cc:460] Webserver started at http://127.2.208.189:37445/ using document root <none> and password file <none>
I20260620 19:00:42.989200 9465 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:42.989243 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:42.990671 9465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:42.991726 9584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:42.992000 9465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:42.992072 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "4e3933b420ab43a1841db9e0d591a974"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "05ce7729a5514ecd1d1d16a301cda82f"
server_key_iv: "35f0c67415efd61e6e77efd868fbff4a"
server_key_version: "encryptionkey@0"
I20260620 19:00:42.992163 9465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.001536 9465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.003604 9465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.008327 9465 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.189:39873
I20260620 19:00:43.008352 9636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.189:39873 every 8 connection(s)
I20260620 19:00:43.008889 9465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
I20260620 19:00:43.010619 9637 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:00:43.011667 9637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Bootstrap starting.
I20260620 19:00:43.021910 9637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:00:43.022258 9637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: Bootstrap complete.
I20260620 19:00:43.022841 9637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } }
I20260620 19:00:43.022993 9637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e3933b420ab43a1841db9e0d591a974, State: Initialized, Role: LEARNER
I20260620 19:00:43.023123 9637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } }
I20260620 19:00:43.023521 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } }
I20260620 19:00:43.023629 9637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:43.023712 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:00:43.026731 9651 catalog_manager.cc:1259] Loaded cluster ID: 6b2e7e34ec74411887986c4e09133417
I20260620 19:00:43.026850 9651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: loading cluster ID for follower catalog manager: success
I20260620 19:00:43.029057 9651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: acquiring CA information for follower catalog manager: success
I20260620 19:00:43.030519 9651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:00:43.087452 9572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:43.087733 9572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:43.087805 9572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:43.091734 9572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:43.091823 9572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:43.091859 9572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:43.091917 9572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:43.091955 9572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:43.097445 9572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.188.keytab
--principal=oryx/127.2.208.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.208.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:43.098523 9572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:43.099593 9572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:43.107838 9656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.107833 9659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.107833 9657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.108279 9572 server_base.cc:1034] running on GCE node
I20260620 19:00:43.108824 9572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:43.109910 9572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:43.111096 9572 hybrid_clock.cc:648] HybridClock initialized: now 1781982043111088 us; error 32 us; skew 500 ppm
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:43.113917 9572 init.cc:377] Logged in from keytab as oryx/127.2.208.188@KRBTEST.COM (short username oryx)
I20260620 19:00:43.114784 9572 webserver.cc:460] Webserver started at http://127.2.208.188:34747/ using document root <none> and password file <none>
I20260620 19:00:43.115345 9572 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.115481 9572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.115720 9572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:43.117692 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/instance:
uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
format_stamp: "Formatted at 2026-06-20 19:00:43 on dist-test-slave-tsm4"
server_key: "d452f074259d6cf92dbc948e8d60ef2a"
server_key_iv: "ac51d1fce22657102a49bf33be0066ff"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.118242 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal/instance:
uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
format_stamp: "Formatted at 2026-06-20 19:00:43 on dist-test-slave-tsm4"
server_key: "d452f074259d6cf92dbc948e8d60ef2a"
server_key_iv: "ac51d1fce22657102a49bf33be0066ff"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.122001 9572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:00:43.124559 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.125855 9572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:43.125984 9572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
format_stamp: "Formatted at 2026-06-20 19:00:43 on dist-test-slave-tsm4"
server_key: "d452f074259d6cf92dbc948e8d60ef2a"
server_key_iv: "ac51d1fce22657102a49bf33be0066ff"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.126093 9572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.165500 9572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.169437 9572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:43.169641 9572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.177614 9572 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.188:33035
I20260620 19:00:43.177928 9718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.188:33035 every 8 connection(s)
I20260620 19:00:43.178574 9572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
I20260620 19:00:43.181376 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:43.185307 9719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:00:43.188319 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9572
I20260620 19:00:43.195123 9192 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.188@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
I20260620 19:00:43.201900 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.187905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.2.208.190:39415)
0620 19:00:43.188109 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.188115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.188978 (+ 863us) server_negotiation.cc:407] Connection header received
0620 19:00:43.190352 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.190356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.190404 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.190500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.191593 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.192433 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.194269 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.194586 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.194692 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.197661 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.197665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.197669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.197701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.199562 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.200094 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.200098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.200100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.200193 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.200609 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.200613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.200614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.200860 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.200947 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.201596 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.201720 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:00:43.202996 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.189337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.2.208.188:33035)
0620 19:00:43.189688 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.189698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.189736 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:00:43.190143 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.190163 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.190386 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.190560 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.191455 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.192717 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.193732 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.194290 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.194449 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.198529 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.198536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.198548 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.198569 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.200392 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.200852 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.200857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.200858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.200952 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.201208 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.201211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.201213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.201463 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.201647 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.201872 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.202252 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":251,"thread_start_us":186,"threads_started":1}
I20260620 19:00:43.202996 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.188231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37550)
0620 19:00:43.188793 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.188859 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.190133 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.190594 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.190600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.191061 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.191454 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.191469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.192605 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.192610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.194131 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.194142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.194737 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.194745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.194866 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.195421 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.195445 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.197486 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.199702 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.199711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.199723 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.199988 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.200293 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.200295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.200297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.200502 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.200958 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.200964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.201307 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.201524 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.201895 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":38,"thread_start_us":126,"threads_started":1}
I20260620 19:00:43.204312 9719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.188@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:43.212329 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.204778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42296)
0620 19:00:43.204886 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.204904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.204975 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.205432 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.205434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.205561 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.205649 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.205653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.206538 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.206540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.207062 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.207068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.207572 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.207575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.207645 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.208050 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.208061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.209278 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.211036 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.211039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.211040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.211208 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.211486 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.211489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.211490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.211566 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.211969 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.211971 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.212040 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.212116 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.212214 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:00:43.212392 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.204977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.2.208.189:39873)
0620 19:00:43.205223 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.205227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.205240 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:43.205280 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.205288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.205327 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.205398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.205765 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.206450 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.207156 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.207482 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.207563 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.209408 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.209411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.209413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.209435 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.210945 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.211288 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.211290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.211291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.211403 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.211669 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.211672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.211673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.211873 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.211959 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.212173 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.212269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20260620 19:00:43.213260 9719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.188@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
I20260620 19:00:43.220459 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.213763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45038)
0620 19:00:43.213867 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.213879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.213941 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.214125 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.214127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.214248 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.214315 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.214319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.214978 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.214980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.215597 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.215605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.215998 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.216001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.216066 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.216437 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.216448 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.217544 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.219177 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.219179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.219181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.219425 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.219669 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.219671 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.219672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.219759 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.220057 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.220059 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.220128 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.220204 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.220303 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:00:43.220472 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.213858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.2.208.188:33035)
0620 19:00:43.213962 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.213970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.213980 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:00:43.214006 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.214008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.214040 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.214099 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.214403 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.214884 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.215690 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.215908 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.215983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.217659 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.217662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.217665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.217686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.219083 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.219529 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.219531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.219532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.219595 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.219831 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.219833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.219834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.219981 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.220051 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.220233 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.220353 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":38}
I20260620 19:00:43.223912 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Bootstrap starting.
I20260620 19:00:43.226359 9719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:43.227082 9719 log.cc:824] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:43.229075 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: No bootstrap required, opened a new log
I20260620 19:00:43.232333 9719 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } }
I20260620 19:00:43.232542 9719 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:43.232626 9719 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fbd45b35e24825a1673ab2e1a17bd2, State: Initialized, Role: FOLLOWER
I20260620 19:00:43.233084 9719 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } }
I20260620 19:00:43.233573 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } } }
I20260620 19:00:43.233707 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:43.234333 9719 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:00:43.239025 9736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:00:43.239130 9736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:00:43.242954 9572 master_runner.cc:428] Detected that this master c6fbd45b35e24825a1673ab2e1a17bd2 is joining an existing cluster
I20260620 19:00:43.243026 9572 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.208.188:33035
I20260620 19:00:43.254390 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.246584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.2.208.188:33035)
0620 19:00:43.246674 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.246678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.246848 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:00:43.246981 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.246984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.247017 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.247059 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.247592 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.248260 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249238 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.249616 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249723 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.251273 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.251276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.251278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.251305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.252755 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.253210 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.253216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.253218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.253296 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.253557 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.253560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.253562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.253727 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.253826 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.254071 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.254236 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":27,"spinlock_wait_cycles":384}
I20260620 19:00:43.254396 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.246341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45054)
0620 19:00:43.246771 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.246786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.246896 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.247153 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.247156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.247355 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.247488 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.247495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.248370 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.248372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.249143 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.249150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.249717 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.249831 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.250466 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.250477 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.250942 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.252859 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.252865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.252867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.253109 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.253381 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.253385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.253388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.253469 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.253815 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.253818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.253897 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.254019 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.254165 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":29,"thread_start_us":72,"threads_started":1}
I20260620 19:00:43.256222 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9063
I20260620 19:00:43.263907 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:42729
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:39415
--webserver_port=40757
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:00:43.272568 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.245905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.2.208.189:39873)
0620 19:00:43.246029 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.246034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.246338 (+ 304us) server_negotiation.cc:407] Connection header received
0620 19:00:43.246510 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.246513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.246572 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.246624 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.247215 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.248039 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.250139 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.250532 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.250608 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.268260 (+ 17652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.268267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.268273 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.268312 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.270207 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.270799 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.270805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.270826 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.270921 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271286 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.271290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.271294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.271620 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.271707 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.272165 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.272329 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:00:43.272718 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.245566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.2.208.190:39415)
0620 19:00:43.245697 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.245703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.245895 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:00:43.246045 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.246052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.246115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.246189 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.246972 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.247829 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249015 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.249366 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.268899 (+ 19467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.268904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.268907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.268941 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.270892 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271410 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.271414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.271418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.271502 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271829 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.271832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.271834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.272043 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.272111 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.272429 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.272561 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:00:43.273221 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.245504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42304)
0620 19:00:43.246247 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.246261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.246401 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.246764 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.246770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.246964 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.247080 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.247087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.248183 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.248186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.250012 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.250020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.250760 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.250763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.250846 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.251268 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.251282 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.267953 (+ 16671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.270371 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.270377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.270382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.270671 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.271053 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.271061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.271168 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.271808 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.271814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.271952 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.272816 (+ 864us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.272986 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":475,"thread_start_us":103,"threads_started":1}
I20260620 19:00:43.273247 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.245504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37566)
0620 19:00:43.245793 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.245810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.245912 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.246356 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.246358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.246599 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.246838 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.246848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.247951 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.247954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.248896 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.248904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.249629 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.249632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.249841 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.268020 (+ 18179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.268039 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.268469 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.271033 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.271038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.271266 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.271624 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.271627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.271629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.271715 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.272180 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.272182 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.272267 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.272916 (+ 649us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.273063 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":129,"spinlock_wait_cycles":1920,"thread_start_us":132,"threads_started":1}
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982041, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.190@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
I20260620 19:00:43.286299 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.277279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45058)
0620 19:00:43.277384 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.277399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.277483 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.277770 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.277773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.277950 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.278050 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.278057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.279040 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.279043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.279962 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.279973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.280554 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.280558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.280655 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.281110 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.281123 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.282608 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.284712 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.284715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.284717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.284957 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.285255 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.285260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.285264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.285354 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.285785 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.285788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.285883 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.285993 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.286132 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:00:43.286314 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.277361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.2.208.188:33035)
0620 19:00:43.277463 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.277467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.277480 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:43.277593 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.277596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.277658 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.277738 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.278164 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.278930 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.280083 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.280444 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.280563 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.282756 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.282763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.282768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.282817 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.284597 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.285056 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.285059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.285061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.285157 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.285447 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.285450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.285452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.285682 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.285779 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.286050 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.286162 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:00:43.287707 9001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.208.188:33035: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
member_type: NON_VOTER
last_known_addr {
host: "127.2.208.188"
port: 33035
}
attrs {
promote: true
}
}
dest_uuid: "e8c6e1cbfd964be289e7b98e721c8c38"
cas_config_opid_index: 5
I20260620 19:00:43.288314 9001 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } }
I20260620 19:00:43.289392 9560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.290902 9560 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.208.188:33035
I20260620 19:00:43.291069 9560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.291167 9560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.292022 9693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:43.292173 9572 master.cc:537] Master@127.2.208.188:33035 shutting down...
I20260620 19:00:43.293450 9693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:00:43.295222 9561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:00:43.297082 8987 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 -> Peer c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188:33035): Couldn't send request to peer c6fbd45b35e24825a1673ab2e1a17bd2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:00:43.297993 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.291646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42308)
0620 19:00:43.293114 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.293128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.293252 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.293658 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.293661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.293677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:43.293759 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.293764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.294835 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.294838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.296145 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.296156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.297589 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.297593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.297684 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.297729 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.297799 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.297866 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356}
I20260620 19:00:43.298347 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.291928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.2.208.189:39873)
0620 19:00:43.292976 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.292981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.293335 (+ 354us) server_negotiation.cc:407] Connection header received
0620 19:00:43.293388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.293391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.293449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.293528 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:43.293897 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.294694 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.296288 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.297260 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.297338 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.297484 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.298137 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.298213 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956}
I20260620 19:00:43.299500 9611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 LEARNER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:00:43.300020 9561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:00:43.301849 9611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.302764 9572 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:43.303051 9572 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:43.303184 9572 tablet_replica.cc:330] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: stopping tablet replica
I20260620 19:00:43.303742 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.303853 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:00:43.304315 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c6e1cbfd964be289e7b98e721c8c38. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: NON_VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: true } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.304435 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:00:43.308573 9572 master.cc:552] Master@127.2.208.188:33035 shutdown complete.
I20260620 19:00:43.308640 9572 master_runner.cc:305] Clearing existing system tablet
I20260620 19:00:43.308892 9572 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.308948 9572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.310329 9572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:00:43.311237 9749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.311559 9572 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.311854 9572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:43.311937 9572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
format_stamp: "Formatted at 2026-06-20 19:00:43 on dist-test-slave-tsm4"
server_key: "d452f074259d6cf92dbc948e8d60ef2a"
server_key_iv: "ac51d1fce22657102a49bf33be0066ff"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.312803 9561 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: attempting to promote NON_VOTER 4e3933b420ab43a1841db9e0d591a974 to VOTER
I20260620 19:00:43.313548 9561 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } }
I20260620 19:00:43.314296 9572 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:43.316679 9572 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:00:43.316816 9572 log.cc:1196] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 19:00:43.317126 9572 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Deleting consensus metadata
I20260620 19:00:43.317329 9572 master_runner.cc:315] Copying system tablet from 127.2.208.190:39415
I20260620 19:00:43.318557 9572 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.208.190:39415
I20260620 19:00:43.320299 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.319922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45066)
0620 19:00:43.320045 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.320168 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:00:43.320511 9611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 LEARNER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 19:00:43.321169 9560 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:00:43.321504 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.321240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45080)
0620 19:00:43.321382 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.321439 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 19:00:43.321733 8987 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 -> Peer c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188:33035): Couldn't send request to peer c6fbd45b35e24825a1673ab2e1a17bd2. Status: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:00:43.323109 9561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.325261 9611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.327044 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.327178 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.327625 9560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.327731 9560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.327997 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.328109 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:43.332643 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.319637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:37568)
0620 19:00:43.323515 (+ 3878us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.323530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.323689 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.324087 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.324090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.324341 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.324491 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.324502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.326003 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.326006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.326889 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.326898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.327681 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.327685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.327946 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.328534 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.328550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.328977 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.331009 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.331012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.331015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.331260 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.331578 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.331581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.331583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.331674 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.332099 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.332102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.332199 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.332306 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.332446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3807,"thread_start_us":93,"threads_started":1}
I20260620 19:00:43.332655 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.319305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.2.208.190:39415)
0620 19:00:43.319524 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.319529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.323612 (+ 4083us) server_negotiation.cc:407] Connection header received
0620 19:00:43.323801 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.323806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.323874 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.323930 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.324716 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.325772 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.327023 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.327344 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.327460 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.329144 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.329148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.329151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.329174 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.330895 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.331387 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.331390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.331392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.331482 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.331774 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.331777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.331779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.331995 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.332101 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.332384 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.332497 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:43.333417 9021 tablet_copy_service.cc:140] P e8c6e1cbfd964be289e7b98e721c8c38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c6fbd45b35e24825a1673ab2e1a17bd2 ({username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:37568)
I20260620 19:00:43.333504 9021 tablet_copy_service.cc:161] P e8c6e1cbfd964be289e7b98e721c8c38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c6fbd45b35e24825a1673ab2e1a17bd2 at {username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:37568: session id = c6fbd45b35e24825a1673ab2e1a17bd2-00000000000000000000000000000000
I20260620 19:00:43.334517 9021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:00:43.335881 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:00:43.338235 9572 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: tablet copy: Starting download of 0 data blocks...
I20260620 19:00:43.338406 9572 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: tablet copy: Starting download of 1 WAL segments...
I20260620 19:00:43.340133 9572 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:00:43.341849 9021 tablet_copy_service.cc:342] P e8c6e1cbfd964be289e7b98e721c8c38: Request end of tablet copy session c6fbd45b35e24825a1673ab2e1a17bd2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:37568
I20260620 19:00:43.341923 9021 tablet_copy_service.cc:433] P e8c6e1cbfd964be289e7b98e721c8c38: ending tablet copy session c6fbd45b35e24825a1673ab2e1a17bd2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c6fbd45b35e24825a1673ab2e1a17bd2
I20260620 19:00:43.349397 9572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:43.351089 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.352135 9758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.352433 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.356230 9572 server_base.cc:1034] running on GCE node
I20260620 19:00:43.356395 9572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:43.356678 9572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:43.357827 9572 hybrid_clock.cc:648] HybridClock initialized: now 1781982043357813 us; error 30 us; skew 500 ppm
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:43.359468 9572 init.cc:377] Logged in from keytab as oryx/127.2.208.188@KRBTEST.COM (short username oryx)
I20260620 19:00:43.359934 9572 webserver.cc:460] Webserver started at http://127.2.208.188:41487/ using document root <none> and password file <none>
I20260620 19:00:43.360118 9572 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.360159 9572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.361553 9572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:43.362540 9767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.362803 9572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:43.362884 9572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
format_stamp: "Formatted at 2026-06-20 19:00:43 on dist-test-slave-tsm4"
server_key: "d452f074259d6cf92dbc948e8d60ef2a"
server_key_iv: "ac51d1fce22657102a49bf33be0066ff"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.362973 9572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.372320 9572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.374651 9572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.380252 9572 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.188:33035
I20260620 19:00:43.380893 9572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
I20260620 19:00:43.381568 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.188:33035 every 8 connection(s)
I20260620 19:00:43.382818 9820 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:00:43.383980 9820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Bootstrap starting.
I20260620 19:00:43.397446 9820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:00:43.397799 9820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Bootstrap complete.
I20260620 19:00:43.398391 9820 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } }
I20260620 19:00:43.398541 9820 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6fbd45b35e24825a1673ab2e1a17bd2, State: Initialized, Role: LEARNER
I20260620 19:00:43.398674 9820 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } }
I20260620 19:00:43.399231 9820 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:43.399376 9823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.399631 9823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:00:43.402989 9834 catalog_manager.cc:1259] Loaded cluster ID: 6b2e7e34ec74411887986c4e09133417
I20260620 19:00:43.403049 9834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: loading cluster ID for follower catalog manager: success
I20260620 19:00:43.405140 9834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: acquiring CA information for follower catalog manager: success
I20260620 19:00:43.406543 9834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:00:43.407271 9326 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
W20260620 19:00:43.415715 9745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:43.415994 9745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:43.416095 9745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:43.420301 9745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:43.420388 9745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:43.420476 9745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:43.425997 9745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:42729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=40757
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:43.427055 9745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:43.428095 9745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:43.437738 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.437741 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.437713 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.438126 9745 server_base.cc:1034] running on GCE node
I20260620 19:00:43.438562 9745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:43.439175 9745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:43.440333 9745 hybrid_clock.cc:648] HybridClock initialized: now 1781982043440327 us; error 57 us; skew 500 ppm
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:43.443218 9745 init.cc:377] Logged in from keytab as oryx/127.2.208.129@KRBTEST.COM (short username oryx)
I20260620 19:00:43.444114 9745 webserver.cc:460] Webserver started at http://127.2.208.129:40757/ using document root <none> and password file <none>
I20260620 19:00:43.444671 9745 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.444768 9745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.448544 9745 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 19:00:43.450606 9849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.451920 9745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:43.452051 9745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "d662bdfa7e804fedace36555fe20d717"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "374ae8b2a4887b625f50e8898134d374"
server_key_iv: "1a14095941cfe430a0ec5ae186359f65"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.452821 9745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.467303 9745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.470734 9745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:43.470932 9745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.471534 9745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:43.472380 9745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:43.472452 9745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.472524 9745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:43.472568 9745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.481776 9745 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:42729
I20260620 19:00:43.481797 9962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:42729 every 8 connection(s)
I20260620 19:00:43.482767 9745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:43.490829 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9745
I20260620 19:00:43.491019 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9197
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:43.497609 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.484615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:33995 (local address 127.2.208.188:33035)
0620 19:00:43.484851 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.484855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.485894 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 19:00:43.486690 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.486694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.486752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.486816 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.488421 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.489448 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.490356 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.490674 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.490779 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.493690 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.493694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.493697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.493722 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.495205 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.495889 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.495893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.495895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.495975 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.496353 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.496358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.496361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.496582 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.496649 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.497287 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.497414 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260620 19:00:43.498515 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.2.208.129:33995)
0620 19:00:43.485816 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.485829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.486483 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.487110 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.487117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.487646 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.488272 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.488281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.489553 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.489558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.490237 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.490245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.490811 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.490815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.490980 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.491843 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.491864 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.493533 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.495331 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.495336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.495352 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.495772 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.496063 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.496065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.496066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.496238 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.496704 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.496709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.497026 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.497238 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.497552 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":133,"spinlock_wait_cycles":37376,"thread_start_us":157,"threads_started":1}
I20260620 19:00:43.499692 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.486125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:36773 (local address 127.2.208.189:39873)
0620 19:00:43.486247 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.486251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.486377 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:00:43.486786 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.486792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.486846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.486923 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.488422 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.489232 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.492694 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.493117 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.493216 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.495912 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.495919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.495925 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.495960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.497732 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.498252 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.498258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.498262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.498348 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.498690 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.498696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.498700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.498909 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.498990 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.499334 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.499503 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:00:43.499918 9963 heartbeater.cc:347] Connected to a master server at 127.2.208.188:33035
I20260620 19:00:43.499970 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:38639
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:39415
--webserver_port=34985
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:00:43.500152 9963 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.500310 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.486013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.2.208.129:36773)
0620 19:00:43.486276 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.486289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.486482 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.487110 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.487117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.487649 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.488275 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.488281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.489345 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.489349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.490486 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.490496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.493424 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.493431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.493566 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.494100 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.494120 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.495771 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.497862 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.497868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.497873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.498139 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.498467 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.498472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.498477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.498571 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.499049 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.499054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.499150 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.499874 (+ 724us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.500021 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":213,"spinlock_wait_cycles":21760,"thread_start_us":92,"threads_started":1}
I20260620 19:00:43.500804 9963 heartbeater.cc:511] Master 127.2.208.188:33035 requested a full tablet report, sending...
I20260620 19:00:43.501324 9965 heartbeater.cc:347] Connected to a master server at 127.2.208.189:39873
I20260620 19:00:43.501405 9965 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.501562 9965 heartbeater.cc:511] Master 127.2.208.189:39873 requested a full tablet report, sending...
I20260620 19:00:43.502736 9776 ts_manager.cc:194] Registered new tserver with Master: d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
I20260620 19:00:43.502738 9593 ts_manager.cc:194] Registered new tserver with Master: d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:43.508669 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.484555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59401 (local address 127.2.208.190:39415)
0620 19:00:43.484702 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.484706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.485663 (+ 957us) server_negotiation.cc:407] Connection header received
0620 19:00:43.486694 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.486697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.486758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.486829 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.488403 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.488939 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.490628 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.490967 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.491058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.505439 (+ 14381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.505442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.505444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.505471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.506931 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.507479 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.507482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.507483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.507558 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.507878 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.507880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.507881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.508052 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.508131 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.508449 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.508544 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:00:43.508849 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:59401)
0620 19:00:43.485475 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.485525 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.486475 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.487109 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.487116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.487654 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.488262 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.488276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.489102 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.489108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.490486 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.490496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.501563 (+ 11067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.501569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.501699 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.503791 (+ 2092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.503802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.505275 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.507051 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.507058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.507062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.507341 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.507664 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.507670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.507673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.507775 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.508147 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.508152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.508274 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.508393 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.508568 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":102,"spinlock_wait_cycles":61312,"thread_start_us":530,"threads_started":1}
I20260620 19:00:43.509512 9964 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:43.509619 9964 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.509790 9964 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
I20260620 19:00:43.510576 9001 ts_manager.cc:194] Re-registered known tserver with Master: d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
I20260620 19:00:43.511140 9001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:59401
W20260620 19:00:43.618988 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:43.619357 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:43.619477 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:43.623737 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:43.623795 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:43.623896 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:43.625423 9460 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
I20260620 19:00:43.629667 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:38639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=34985
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:43.630671 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:43.631665 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:43.641372 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.641372 9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.641516 9977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.641901 9971 server_base.cc:1034] running on GCE node
I20260620 19:00:43.642356 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:43.643040 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:43.644212 9971 hybrid_clock.cc:648] HybridClock initialized: now 1781982043644116 us; error 112 us; skew 500 ppm
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:43.646809 9971 init.cc:377] Logged in from keytab as oryx/127.2.208.130@KRBTEST.COM (short username oryx)
I20260620 19:00:43.647708 9971 webserver.cc:460] Webserver started at http://127.2.208.130:34985/ using document root <none> and password file <none>
I20260620 19:00:43.648218 9971 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.648288 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.651999 9971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:43.653909 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.655148 9971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:43.655275 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "42bc808a023544749fd7aaf831414970"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "2377382029f74267a61d6005f23f8a77"
server_key_iv: "39a4fab92c4f09a928078192374340fd"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.655859 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.692059 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.695181 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:43.695456 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.696064 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:43.697008 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:43.697070 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.697122 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:43.697154 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.699054 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.695125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45092)
0620 19:00:43.695344 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.695360 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.695474 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.695774 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.695777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.695794 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:43.695895 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.695900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.696759 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.696762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.697932 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.697946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.698617 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.698622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.698723 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.698753 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.698841 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.698923 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 19:00:43.699041 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.695250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.2.208.188:33035)
0620 19:00:43.695421 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.695425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.695441 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:00:43.695601 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.695611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.695664 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.695763 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:43.696007 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.696615 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.698076 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.698494 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.698582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.698731 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.698881 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.698943 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:00:43.700088 9786 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 19:00:43.700639 9823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c6e1cbfd964be289e7b98e721c8c38. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.700820 9823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:00:43.706848 9971 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:38639
I20260620 19:00:43.706867 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:38639 every 8 connection(s)
I20260620 19:00:43.707881 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:43.709043 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 9971
I20260620 19:00:43.709201 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9331
I20260620 19:00:43.717860 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:44091
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:39415
--webserver_port=33647
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
I20260620 19:00:43.726748 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.710630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:37797 (local address 127.2.208.188:33035)
0620 19:00:43.710756 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.710761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.712133 (+ 1372us) server_negotiation.cc:407] Connection header received
0620 19:00:43.712825 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.712831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.712896 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.712952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.714781 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715690 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.717680 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718028 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.722918 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.722923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.722927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.722957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.724874 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725485 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.725488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.725490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.725564 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725868 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.725871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.725872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.726054 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.726120 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.726463 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.726591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:00:43.726748 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.711818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:55689 (local address 127.2.208.189:39873)
0620 19:00:43.712566 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.712571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.712586 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:43.712853 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.712857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.712906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.712959 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.714775 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715616 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.717710 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718058 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718122 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.722301 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.722305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.722308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.722336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.724463 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.724992 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.724996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.724998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.725093 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725485 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.725488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.725490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.725722 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.725793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.726463 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.726581 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661}
I20260620 19:00:43.726863 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.711712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:44685 (local address 127.2.208.190:39415)
0620 19:00:43.711829 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.711833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.711850 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:43.713487 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.713491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.713560 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.713730 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.714775 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715617 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.717710 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718049 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718122 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.721875 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.721882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.721887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.721923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.724184 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.724932 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.724938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.724942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.725054 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725517 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.725522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.725523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.725775 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.725860 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.726501 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.726646 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:00:43.728335 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.711520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.2.208.130:55689)
0620 19:00:43.712333 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.712354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.712750 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.713282 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.713288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.713955 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.714610 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.714628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715796 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.715800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.717572 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.717582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718398 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.718590 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.719528 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.719548 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.722123 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.724603 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.724607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.724612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.724879 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.725187 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.725197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.725376 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.725871 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.725878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.726183 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.727201 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.727316 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":502,"spinlock_wait_cycles":14208,"thread_start_us":71,"threads_started":1}
I20260620 19:00:43.728335 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.710565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.2.208.130:37797)
0620 19:00:43.712050 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.712064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.712587 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.713285 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.713291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.713856 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.714620 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.714633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715838 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.715844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.717547 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.717557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718313 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.718578 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.719607 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.719621 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.722762 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.724998 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.725004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.725376 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.725670 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.725674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.725767 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.726188 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.726190 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.726257 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.726901 (+ 644us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.727246 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"mutex_wait_us":302,"thread_start_us":127,"threads_started":1}
I20260620 19:00:43.728336 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.710090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:44685)
0620 19:00:43.711154 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.711205 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.712570 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.713966 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.713968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.714118 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.714610 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.714628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.715796 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.715800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.717572 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.717582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.718387 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.718393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.718603 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.719458 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.719482 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.721691 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.724331 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.724339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.724359 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.724804 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.725187 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.725193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.725196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.725392 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.725946 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.725950 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.726189 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.727626 (+ 1437us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.727801 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":528,"spinlock_wait_cycles":51456,"thread_start_us":699,"threads_started":1}
I20260620 19:00:43.729635 10101 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:43.729681 10102 heartbeater.cc:347] Connected to a master server at 127.2.208.189:39873
I20260620 19:00:43.729899 10102 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.729900 10101 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.729925 10100 heartbeater.cc:347] Connected to a master server at 127.2.208.188:33035
I20260620 19:00:43.730077 10100 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.730554 10100 heartbeater.cc:511] Master 127.2.208.188:33035 requested a full tablet report, sending...
I20260620 19:00:43.730561 10101 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
I20260620 19:00:43.730567 10102 heartbeater.cc:511] Master 127.2.208.189:39873 requested a full tablet report, sending...
I20260620 19:00:43.731731 9001 ts_manager.cc:194] Re-registered known tserver with Master: 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
I20260620 19:00:43.731802 9776 ts_manager.cc:194] Registered new tserver with Master: 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
I20260620 19:00:43.732467 9001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:44685
I20260620 19:00:43.733098 9593 ts_manager.cc:194] Registered new tserver with Master: 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
W20260620 19:00:43.834290 10107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:43.834573 10107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:43.834643 10107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:43.838831 10107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:43.838907 10107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:43.839000 10107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:43.844344 10107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:44091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=33647
--tserver_master_addrs=127.2.208.190:39415,127.2.208.189:39873,127.2.208.188:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:43.845382 10107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:43.846395 10107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:43.855605 10115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.855691 10107 server_base.cc:1034] running on GCE node
W20260620 19:00:43.855614 10112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:43.855633 10113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:43.856343 10107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:43.856998 10107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:43.858191 10107 hybrid_clock.cc:648] HybridClock initialized: now 1781982043858161 us; error 41 us; skew 500 ppm
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:43.861069 10107 init.cc:377] Logged in from keytab as oryx/127.2.208.131@KRBTEST.COM (short username oryx)
I20260620 19:00:43.861982 10107 webserver.cc:460] Webserver started at http://127.2.208.131:33647/ using document root <none> and password file <none>
I20260620 19:00:43.862591 10107 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:43.862692 10107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:43.866411 10107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:43.868485 10122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.869684 10107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:43.869810 10107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e596d9f6904443e19f9dca54132ec024"
format_stamp: "Formatted at 2026-06-20 19:00:42 on dist-test-slave-tsm4"
server_key: "b33968b298dfd0f22363bd89a2770f1e"
server_key_iv: "9b00a4ab40bfe8e656d32f1c9b12a144"
server_key_version: "encryptionkey@0"
I20260620 19:00:43.870337 10107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:43.898006 10107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:43.901309 10107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:43.901525 10107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:43.902132 10107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:43.902971 10107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:43.903040 10107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.903107 10107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:43.903158 10107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:43.912681 10107 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:44091
I20260620 19:00:43.912700 10235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:44091 every 8 connection(s)
I20260620 19:00:43.913671 10107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:43.914603 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 10107
I20260620 19:00:43.928366 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.917188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.2.208.188:33035)
0620 19:00:43.917694 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.917698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.917712 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:43.917989 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.917995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.918052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.918135 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.919030 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.919909 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.921089 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.921431 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.921524 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.923582 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.923588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.923592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.923627 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.925674 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.926376 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.926380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.926382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.926465 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.927003 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.927006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.927008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.927213 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.927336 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.928030 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.928159 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":54,"threads_started":1}
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.188@KRBTEST.COM
I20260620 19:00:43.932468 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.917124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.2.208.190:39415)
0620 19:00:43.919155 (+ 2031us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.919159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.919171 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:43.919206 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.919208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.919246 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.919327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.920120 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.920917 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.921995 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.922373 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.922475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.924464 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.924470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.924476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.924513 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.928214 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.929774 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.929777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.929779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.929866 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.930695 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.930701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.930705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.930909 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.931052 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.932174 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.932299 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1964,"thread_start_us":64,"threads_started":1}
Jun 20 19:00:43 dist-test-slave-tsm4 krb5kdc[8956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982043, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.189@KRBTEST.COM
I20260620 19:00:43.935917 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.916434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:52197 (local address 127.2.208.188:33035)
0620 19:00:43.916577 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.916584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.922743 (+ 6159us) server_negotiation.cc:407] Connection header received
0620 19:00:43.922887 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.922890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.922935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.922982 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.923606 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.924401 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.926740 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.927123 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.927193 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.931023 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.931028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.931032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.931057 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.933392 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.934589 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.934593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.934594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.934700 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.935004 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.935007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.935008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.935231 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.935312 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.935648 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.935780 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:00:43.935917 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.915496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:37419 (local address 127.2.208.190:39415)
0620 19:00:43.915663 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.915667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.917261 (+ 1594us) server_negotiation.cc:407] Connection header received
0620 19:00:43.918490 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.918494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.918542 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.918590 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.920229 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.920983 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.925645 (+ 4662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.925979 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.926044 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.931452 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.931457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.931460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.931487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.933286 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.934038 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.934044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.934049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.934147 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.934602 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.934605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.934607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.934841 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.934925 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.935659 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.935770 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:00:43.937680 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.915987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.2.208.131:52197)
0620 19:00:43.922676 (+ 6689us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.922688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.922796 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.923125 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.923128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.923331 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.923483 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.923489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.924565 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.924568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.925478 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.925485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.927326 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.927330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.927448 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.929290 (+ 1842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.929305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.930836 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.933497 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.933502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.933507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.934167 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.934810 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.934813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.934815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.934905 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.935391 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.935393 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.935470 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.936060 (+ 590us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.936543 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6607,"mutex_wait_us":110,"spinlock_wait_cycles":39296,"thread_start_us":143,"threads_started":1}
I20260620 19:00:43.937680 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.915987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:37419)
0620 19:00:43.917126 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.917180 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.918268 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.918964 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.918972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.919434 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.920073 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.920090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.921136 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.921139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.922474 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.922484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.926234 (+ 3750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.926237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.926454 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.927854 (+ 1400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.927875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.931285 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.933425 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.933435 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.933458 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.933918 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.934268 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.934273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.934277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.934473 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.934987 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.934996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.935334 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.936092 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.936600 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":144,"thread_start_us":124,"threads_started":1}
I20260620 19:00:43.938522 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.918579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.2.208.131:35095)
0620 19:00:43.923094 (+ 4515us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.923109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.925218 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.925919 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.925923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.926134 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:43.926265 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.926272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.928750 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.928756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.929647 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.929656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.931810 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.931814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.931930 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.932447 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:43.932466 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:43.935027 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:43.937059 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.937062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.937064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.937289 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.937563 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:43.937566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:43.937568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:43.937644 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:43.938059 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:43.938062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:43.938142 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.938237 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.938364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4427,"thread_start_us":85,"threads_started":1}
I20260620 19:00:43.935590 9001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:00:43.938970 10237 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39415
I20260620 19:00:43.938993 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.918893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:35095 (local address 127.2.208.189:39873)
0620 19:00:43.924977 (+ 6084us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.924983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.925000 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:43.925351 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.925357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.925415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.925499 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.926390 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.927177 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.930170 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.930492 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.930587 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.935184 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.935187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.935190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.935216 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.936932 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.937391 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.937395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.937396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.937471 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.937735 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.937738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.937740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.937925 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.938007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.938716 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.938835 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5951,"thread_start_us":75,"threads_started":1}
I20260620 19:00:43.939231 10237 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.939798 10237 heartbeater.cc:511] Master 127.2.208.190:39415 requested a full tablet report, sending...
W20260620 19:00:43.940460 9001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:43.940616 10236 heartbeater.cc:347] Connected to a master server at 127.2.208.188:33035
I20260620 19:00:43.940708 10236 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.940828 9000 ts_manager.cc:194] Re-registered known tserver with Master: e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091)
I20260620 19:00:43.940876 10236 heartbeater.cc:511] Master 127.2.208.188:33035 requested a full tablet report, sending...
I20260620 19:00:43.941471 10238 heartbeater.cc:347] Connected to a master server at 127.2.208.189:39873
I20260620 19:00:43.941565 10238 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:43.941542 9000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:37419
I20260620 19:00:43.941601 9776 ts_manager.cc:194] Registered new tserver with Master: e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091)
I20260620 19:00:43.941711 10238 heartbeater.cc:511] Master 127.2.208.189:39873 requested a full tablet report, sending...
I20260620 19:00:43.942384 9593 ts_manager.cc:194] Registered new tserver with Master: e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091)
I20260620 19:00:43.944139 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: NON_VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: true } } }
I20260620 19:00:43.944380 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.947942 9756 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: attempting to promote NON_VOTER c6fbd45b35e24825a1673ab2e1a17bd2 to VOTER
I20260620 19:00:43.947999 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.916719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.2.208.189:39873)
0620 19:00:43.916838 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.916844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.917144 (+ 300us) server_negotiation.cc:407] Connection header received
0620 19:00:43.917358 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.917362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.917414 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.917467 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:43.929237 (+ 11770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.929957 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.932929 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.933241 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.933312 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.943615 (+ 10303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:43.943621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:43.943623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:43.943651 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:43.945651 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.946159 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.946162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.946167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.946260 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:43.946589 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:43.946592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:43.946593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:43.946815 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:43.946961 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.947711 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.947824 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:43.948267 9756 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:43.949623 9611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:00:43.949753 9786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 LEARNER]: Refusing update from remote peer e8c6e1cbfd964be289e7b98e721c8c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:00:43.950143 10255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:00:43.950431 10255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:00:43.952384 9756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.952965 9611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.953024 9786 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.953912 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.954006 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.954102 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.954182 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.954083 10254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.954295 10254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:43.955065 9823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.955173 9823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:43.955487 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:43.955610 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:43.962363 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.957194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44091 (local address 127.0.0.1:35320)
0620 19:00:43.957315 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.957330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.957401 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.958248 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.958251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.958271 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:43.958382 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.958388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.959803 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.959806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.961029 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.961039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.961987 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.961990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.962085 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.962110 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.962174 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.962234 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:00:43.962442 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.957378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.2.208.131:44091)
0620 19:00:43.957706 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.957720 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.957743 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:00:43.957826 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.957833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.958079 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.958284 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:43.958503 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.959674 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.961183 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.961884 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.961990 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.962142 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.962201 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.962288 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260620 19:00:43.962845 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.957742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:42729 (local address 127.0.0.1:45342)
0620 19:00:43.958014 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.958031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.958445 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.959400 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.959404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.959421 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:43.959530 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.959536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.960548 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.960554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.961477 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.961486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.962529 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.962532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.962640 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.962660 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.962704 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.962752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
I20260620 19:00:43.963907 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.958146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.2.208.129:42729)
0620 19:00:43.958663 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.958678 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.958699 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:00:43.958982 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.958988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.959234 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.959406 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:43.959635 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.960439 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.961607 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.962347 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.963219 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.963361 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.963677 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.963773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":105,"threads_started":1}
I20260620 19:00:43.965368 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.958208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.2.208.130:38639)
0620 19:00:43.958754 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:43.958766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:43.959097 (+ 331us) server_negotiation.cc:407] Connection header received
0620 19:00:43.959177 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:43.959183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:43.959400 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:43.959625 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:43.960020 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.961119 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.963690 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.964354 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.964474 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.964621 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:43.965185 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 19:00:43.965263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":90,"threads_started":1}
I20260620 19:00:43.965709 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:43.957810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38639 (local address 127.0.0.1:51390)
0620 19:00:43.958889 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:43.958899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:43.959006 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:43.959803 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:43.959806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:43.959823 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:43.959901 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.959907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.961761 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.961767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.963585 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:43.963596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:43.964822 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:43.964825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:43.964914 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:43.964939 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:00:43.965525 (+ 586us) client_negotiation.cc:240] Negotiation successful
0620 19:00:43.965609 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":116,"threads_started":1}
I20260620 19:00:43.965906 10170 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 (DEFAULT_TABLE table=test-table [id=cde0e37a5a4d44c0ac999ef8a2e004ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:43.967060 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4056f8ce84741b6f5443e48ddc128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:43.966706 9897 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 (DEFAULT_TABLE table=test-table [id=cde0e37a5a4d44c0ac999ef8a2e004ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:43.967772 9897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4056f8ce84741b6f5443e48ddc128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:43.967886 10034 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 (DEFAULT_TABLE table=test-table [id=cde0e37a5a4d44c0ac999ef8a2e004ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:43.968732 10034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4056f8ce84741b6f5443e48ddc128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:43.974840 10269 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Bootstrap starting.
I20260620 19:00:43.975890 10270 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Bootstrap starting.
I20260620 19:00:43.976926 10269 tablet_bootstrap.cc:654] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:43.977648 10269 log.cc:824] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:43.978058 10270 tablet_bootstrap.cc:654] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:43.978715 10270 log.cc:824] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:43.979427 10271 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Bootstrap starting.
I20260620 19:00:43.979705 10269 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: No bootstrap required, opened a new log
I20260620 19:00:43.979887 10269 ts_tablet_manager.cc:1397] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:00:43.980660 10270 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: No bootstrap required, opened a new log
I20260620 19:00:43.980840 10270 ts_tablet_manager.cc:1397] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:43.981205 10271 tablet_bootstrap.cc:654] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:43.981817 10271 log.cc:824] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:43.983455 10271 tablet_bootstrap.cc:492] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: No bootstrap required, opened a new log
I20260620 19:00:43.983654 10271 ts_tablet_manager.cc:1397] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:00:43.984526 10269 raft_consensus.cc:348] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.984810 10269 raft_consensus.cc:374] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:43.984871 10269 raft_consensus.cc:729] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bc808a023544749fd7aaf831414970, State: Initialized, Role: FOLLOWER
I20260620 19:00:43.985330 10269 consensus_queue.cc:260] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.985510 10270 raft_consensus.cc:348] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.985749 10270 raft_consensus.cc:374] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:43.985793 10270 raft_consensus.cc:729] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d662bdfa7e804fedace36555fe20d717, State: Initialized, Role: FOLLOWER
I20260620 19:00:43.986182 10270 consensus_queue.cc:260] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.986327 10269 ts_tablet_manager.cc:1428] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:00:43.986734 10101 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
I20260620 19:00:43.987053 10270 ts_tablet_manager.cc:1428] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:00:43.989262 10271 raft_consensus.cc:348] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.989483 9964 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
I20260620 19:00:43.989544 10271 raft_consensus.cc:374] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:43.989602 10271 raft_consensus.cc:729] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e596d9f6904443e19f9dca54132ec024, State: Initialized, Role: FOLLOWER
I20260620 19:00:43.990036 10271 consensus_queue.cc:260] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:43.990921 10237 heartbeater.cc:503] Master 127.2.208.190:39415 was elected leader, sending a full tablet report...
I20260620 19:00:43.990949 10271 ts_tablet_manager.cc:1428] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:00:44.012394 10276 raft_consensus.cc:482] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:44.012596 10276 raft_consensus.cc:504] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:44.013839 10276 leader_election.cc:288] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091), 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
I20260620 19:00:44.018508 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.014319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59521 (local address 127.2.208.131:44091)
0620 19:00:44.014483 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.014487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.014538 (+ 51us) server_negotiation.cc:407] Connection header received
0620 19:00:44.014763 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.014769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.014825 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.014928 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:44.015197 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.015771 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.017290 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.017875 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.017960 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.018008 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.018309 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.018384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 19:00:44.018630 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.014268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:40429 (local address 127.2.208.130:38639)
0620 19:00:44.014410 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.014415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.014731 (+ 316us) server_negotiation.cc:407] Connection header received
0620 19:00:44.014961 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.014965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.015018 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.015065 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:44.015457 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.016252 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.017526 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.018092 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.018239 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.018283 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.018450 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.018516 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:00:44.018642 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.014042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38639 (local address 127.2.208.129:40429)
0620 19:00:44.014640 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:44.014654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:44.014798 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:44.015196 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:44.015199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:44.015214 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:44.015311 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:44.015316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.016364 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.016368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:44.017402 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:44.017410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.018225 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.018230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:44.018320 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.018341 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:00:44.018436 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:00:44.018507 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":384,"thread_start_us":71,"threads_started":1}
I20260620 19:00:44.019066 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.014042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44091 (local address 127.2.208.129:59521)
0620 19:00:44.014450 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:44.014465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:44.014620 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:44.014952 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:44.014955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:44.014971 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:44.015086 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:44.015091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.015918 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.015923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:44.017159 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:44.017169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.018072 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.018075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:44.018182 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.018204 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:00:44.018865 (+ 661us) client_negotiation.cc:240] Negotiation successful
0620 19:00:44.018938 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":64,"threads_started":1}
I20260620 19:00:44.019256 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4056f8ce84741b6f5443e48ddc128" candidate_uuid: "d662bdfa7e804fedace36555fe20d717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc808a023544749fd7aaf831414970" is_pre_election: true
I20260620 19:00:44.019591 10054 raft_consensus.cc:2454] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d662bdfa7e804fedace36555fe20d717 in term 0.
I20260620 19:00:44.019620 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4056f8ce84741b6f5443e48ddc128" candidate_uuid: "d662bdfa7e804fedace36555fe20d717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e596d9f6904443e19f9dca54132ec024" is_pre_election: true
I20260620 19:00:44.019896 10190 raft_consensus.cc:2454] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d662bdfa7e804fedace36555fe20d717 in term 0.
I20260620 19:00:44.020085 9852 leader_election.cc:302] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42bc808a023544749fd7aaf831414970, d662bdfa7e804fedace36555fe20d717; no voters:
I20260620 19:00:44.020356 10276 raft_consensus.cc:2781] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:44.020437 10276 raft_consensus.cc:482] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:44.020479 10276 raft_consensus.cc:3037] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:44.021468 10276 raft_consensus.cc:504] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:44.021824 10276 leader_election.cc:288] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [CANDIDATE]: Term 1 election: Requested vote from peers e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091), 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639)
I20260620 19:00:44.022141 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4056f8ce84741b6f5443e48ddc128" candidate_uuid: "d662bdfa7e804fedace36555fe20d717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e596d9f6904443e19f9dca54132ec024"
I20260620 19:00:44.022194 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4056f8ce84741b6f5443e48ddc128" candidate_uuid: "d662bdfa7e804fedace36555fe20d717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc808a023544749fd7aaf831414970"
I20260620 19:00:44.022270 10190 raft_consensus.cc:3037] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:44.022295 10054 raft_consensus.cc:3037] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:44.023972 10054 raft_consensus.cc:2454] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d662bdfa7e804fedace36555fe20d717 in term 1.
I20260620 19:00:44.023972 10190 raft_consensus.cc:2454] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d662bdfa7e804fedace36555fe20d717 in term 1.
I20260620 19:00:44.024227 9852 leader_election.cc:302] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42bc808a023544749fd7aaf831414970, d662bdfa7e804fedace36555fe20d717; no voters:
I20260620 19:00:44.024408 10276 raft_consensus.cc:2781] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:44.024657 10276 raft_consensus.cc:686] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 1 LEADER]: Becoming Leader. State: Replica: d662bdfa7e804fedace36555fe20d717, State: Running, Role: LEADER
I20260620 19:00:44.025003 10276 consensus_queue.cc:237] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:44.028390 8999 catalog_manager.cc:5495] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 reported cstate change: term changed from 0 to 1, leader changed from <none> to d662bdfa7e804fedace36555fe20d717 (127.2.208.129). New cstate: current_term: 1 leader_uuid: "d662bdfa7e804fedace36555fe20d717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } health_report { overall_health: HEALTHY } } }
I20260620 19:00:44.040855 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.037315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.2.208.129:42729)
0620 19:00:44.037433 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.037438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.037471 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:00:44.037654 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.037658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.037715 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.037816 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:44.038053 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.038578 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.039453 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.039679 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.039786 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.040071 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:44.040479 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:44.040608 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.040668 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.040717 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:44.046470 10054 raft_consensus.cc:1261] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Refusing update from remote peer d662bdfa7e804fedace36555fe20d717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:44.046470 10190 raft_consensus.cc:1261] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Refusing update from remote peer d662bdfa7e804fedace36555fe20d717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:44.047153 10280 consensus_queue.cc:1035] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:44.047416 10276 consensus_queue.cc:1035] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:44.054404 10284 mvcc.cc:205] Tried to move back new op lower bound from 7298998452407791616 to 7298998452328939520. Current Snapshot: MvccSnapshot[applied={T|T < 7298998452407791616}]
I20260620 19:00:44.061591 10285 mvcc.cc:205] Tried to move back new op lower bound from 7298998452407791616 to 7298998452328939520. Current Snapshot: MvccSnapshot[applied={T|T < 7298998452407791616}]
I20260620 19:00:44.063424 9000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:44.063584 9000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:44.067852 9000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c6e1cbfd964be289e7b98e721c8c38: Sending DeleteTablet for 3 replicas of tablet d1b4056f8ce84741b6f5443e48ddc128
I20260620 19:00:44.068589 10034 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:44 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:51390
I20260620 19:00:44.068672 10170 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:44 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:35320
I20260620 19:00:44.068778 9897 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b4056f8ce84741b6f5443e48ddc128 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:44 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:45342
I20260620 19:00:44.069159 10294 tablet_replica.cc:330] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: stopping tablet replica
I20260620 19:00:44.069495 10294 raft_consensus.cc:2229] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:44.069829 10294 raft_consensus.cc:2258] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:44.069957 10295 tablet_replica.cc:330] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: stopping tablet replica
I20260620 19:00:44.070210 10295 raft_consensus.cc:2229] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:44.070544 10295 raft_consensus.cc:2258] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:44.071168 10293 tablet_replica.cc:330] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: stopping tablet replica
I20260620 19:00:44.071422 10294 ts_tablet_manager.cc:1905] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:44.071452 10293 raft_consensus.cc:2229] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:44.071669 10293 raft_consensus.cc:2258] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:44.071949 10295 ts_tablet_manager.cc:1905] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:44.072261 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 8969
I20260620 19:00:44.075102 10295 ts_tablet_manager.cc:1918] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:44.075264 10295 log.cc:1196] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/d1b4056f8ce84741b6f5443e48ddc128
I20260620 19:00:44.075316 10294 ts_tablet_manager.cc:1918] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:44.075451 10294 log.cc:1196] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/d1b4056f8ce84741b6f5443e48ddc128
I20260620 19:00:44.075803 10294 ts_tablet_manager.cc:1939] T d1b4056f8ce84741b6f5443e48ddc128 P e596d9f6904443e19f9dca54132ec024: Deleting consensus metadata
I20260620 19:00:44.075825 10295 ts_tablet_manager.cc:1939] T d1b4056f8ce84741b6f5443e48ddc128 P d662bdfa7e804fedace36555fe20d717: Deleting consensus metadata
W20260620 19:00:44.079746 9850 connection.cc:535] server connection from 127.0.0.1:45342 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:00:44.080135 10124 connection.cc:535] server connection from 127.0.0.1:35320 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:00:44.082260 10293 ts_tablet_manager.cc:1905] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:44.084791 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.071715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.2.208.189:39873)
0620 19:00:44.071830 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.071834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.075431 (+ 3597us) server_negotiation.cc:407] Connection header received
0620 19:00:44.075717 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.075721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.075779 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.075837 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:44.076881 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.077449 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.078249 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.078547 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.078652 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.081118 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:44.081122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:44.081125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:44.081152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:44.082905 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.083440 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.083443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.083445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.083531 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.083857 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.083861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.083862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.084096 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:44.084242 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.084496 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.084624 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:00:44.084978 10293 ts_tablet_manager.cc:1918] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:44.085069 10293 log.cc:1196] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/d1b4056f8ce84741b6f5443e48ddc128
I20260620 19:00:44.085398 10293 ts_tablet_manager.cc:1939] T d1b4056f8ce84741b6f5443e48ddc128 P 42bc808a023544749fd7aaf831414970: Deleting consensus metadata
W20260620 19:00:44.086426 9989 connection.cc:441] server connection from 127.0.0.1:51390 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51390 (request call id 1) could send its response
I20260620 19:00:44.094282 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.072027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.2.208.188:33035)
0620 19:00:44.072131 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.072135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.084965 (+ 12830us) server_negotiation.cc:407] Connection header received
0620 19:00:44.085305 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.085309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.085360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.085409 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:44.086270 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.087020 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.088087 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.088318 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.088432 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.089772 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:44.089776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:44.089779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:44.089804 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:44.091420 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.092622 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.092625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.092627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.092831 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.093212 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.093218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.093221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.093453 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:44.093574 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.093969 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.094087 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:00:44.096603 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.087092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.2.208.189:39873)
0620 19:00:44.087206 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.087210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.087553 (+ 343us) server_negotiation.cc:407] Connection header received
0620 19:00:44.087703 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.087706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.087753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.087799 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:44.088746 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.089504 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.090685 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.091021 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.091112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.092606 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:44.092610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:44.092612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:44.092637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:44.094480 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.094973 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.094976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.094978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.095062 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.095550 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.095557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.095558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.095986 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:44.096124 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.096365 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.096469 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:00:44.099825 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:44.087638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.2.208.188:33035)
0620 19:00:44.087740 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:44.087744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:44.091421 (+ 3677us) server_negotiation.cc:407] Connection header received
0620 19:00:44.091576 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:44.091579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:44.091622 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:44.091668 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:44.092500 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.093309 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.094836 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:44.095179 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:44.095248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:44.096784 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:44.096788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:44.096790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:44.096819 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:44.098231 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.098654 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.098657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.098658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.098732 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:44.099000 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:44.099003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:44.099004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:44.099146 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:44.099239 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:44.099487 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:00:44.099625 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:00:45.058858 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.058468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:48247)
0620 19:00:45.058681 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.058770 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
W20260620 19:00:45.059159 10237 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:39415 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.063434 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.062968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:36491)
0620 19:00:45.063218 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.063313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
W20260620 19:00:45.063846 10101 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:39415 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.064397 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.064066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:49969)
0620 19:00:45.064263 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.064327 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":52,"threads_started":1}
W20260620 19:00:45.064635 9964 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:39415 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.568428 10306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8c6e1cbfd964be289e7b98e721c8c38)
I20260620 19:00:45.568635 10306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:45.568806 10307 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e8c6e1cbfd964be289e7b98e721c8c38)
I20260620 19:00:45.568981 10307 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:45.569747 10306 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188:33035)
I20260620 19:00:45.569870 10307 leader_election.cc:288] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189:39873)
I20260620 19:00:45.570683 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.569916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38042)
0620 19:00:45.570491 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.570576 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20260620 19:00:45.570847 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.570032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38050)
0620 19:00:45.570702 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.570772 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":87,"threads_started":1}
W20260620 19:00:45.570844 9586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
W20260620 19:00:45.570969 9769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.571786 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.571537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38066)
0620 19:00:45.571673 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.571721 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 19:00:45.571918 9586 leader_election.cc:334] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.572068 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.571535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38056)
0620 19:00:45.571813 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.571946 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
W20260620 19:00:45.572175 9769 leader_election.cc:334] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.574465 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.569948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33035 (local address 127.0.0.1:45130)
0620 19:00:45.570467 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.570484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:45.570597 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:45.570982 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:45.570988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:45.571007 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:45.571111 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:45.571116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.572216 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.572219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:45.573398 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:45.573407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.574093 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.574100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:45.574208 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:45.574238 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:00:45.574306 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:00:45.574374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260620 19:00:45.574515 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.570334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.2.208.188:33035)
0620 19:00:45.570677 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:45.570684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:45.570696 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:45.570770 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:45.570773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:45.570822 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:45.570904 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:45.571272 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.572059 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.573556 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.573980 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.574062 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:45.574107 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:45.574339 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:00:45.574410 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
I20260620 19:00:45.575032 9786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3933b420ab43a1841db9e0d591a974" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" is_pre_election: true
I20260620 19:00:45.575240 9786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3933b420ab43a1841db9e0d591a974 in term 1.
I20260620 19:00:45.575399 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.570212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.2.208.189:39873)
0620 19:00:45.570531 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:45.570536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:45.571407 (+ 871us) server_negotiation.cc:407] Connection header received
0620 19:00:45.571596 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:45.571599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:45.571641 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:45.571706 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:45.572091 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.572896 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.574181 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.574868 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.574983 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:45.575032 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:45.575205 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:00:45.575265 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":251,"thread_start_us":93,"threads_started":1}
I20260620 19:00:45.575408 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.570425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:39873 (local address 127.0.0.1:42344)
0620 19:00:45.571303 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.571317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:45.571481 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:45.571846 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:45.571849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:45.571869 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:45.571963 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:45.571970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.573024 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.573051 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:45.574054 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:45.574066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:45.574982 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:45.574986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:45.575072 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:45.575092 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:00:45.575192 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:00:45.575256 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":43,"threads_started":1}
I20260620 19:00:45.575734 9587 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e3933b420ab43a1841db9e0d591a974, c6fbd45b35e24825a1673ab2e1a17bd2; no voters: e8c6e1cbfd964be289e7b98e721c8c38
I20260620 19:00:45.575767 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e3933b420ab43a1841db9e0d591a974" is_pre_election: true
I20260620 19:00:45.575944 9611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6fbd45b35e24825a1673ab2e1a17bd2 in term 1.
I20260620 19:00:45.576040 10306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:00:45.576098 10306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e8c6e1cbfd964be289e7b98e721c8c38)
I20260620 19:00:45.576155 10306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:00:45.576265 9770 leader_election.cc:302] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e3933b420ab43a1841db9e0d591a974, c6fbd45b35e24825a1673ab2e1a17bd2; no voters: e8c6e1cbfd964be289e7b98e721c8c38
I20260620 19:00:45.576471 10307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:00:45.576531 10307 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e8c6e1cbfd964be289e7b98e721c8c38)
I20260620 19:00:45.576570 10307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:00:45.577533 10306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:45.577780 10307 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:45.577927 10306 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 election: Requested vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), c6fbd45b35e24825a1673ab2e1a17bd2 (127.2.208.188:33035)
I20260620 19:00:45.578168 10307 leader_election.cc:288] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 election: Requested vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189:39873)
I20260620 19:00:45.578285 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.578050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38076)
0620 19:00:45.578168 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.578228 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 19:00:45.578425 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e3933b420ab43a1841db9e0d591a974"
I20260620 19:00:45.578572 9611 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c6fbd45b35e24825a1673ab2e1a17bd2 in current term 2: Already voted for candidate 4e3933b420ab43a1841db9e0d591a974 in this term.
I20260620 19:00:45.578562 9786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3933b420ab43a1841db9e0d591a974" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2"
I20260620 19:00:45.578696 9786 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e3933b420ab43a1841db9e0d591a974 in current term 2: Already voted for candidate c6fbd45b35e24825a1673ab2e1a17bd2 in this term.
I20260620 19:00:45.578923 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.578710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38098)
0620 19:00:45.578828 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.578871 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:00:45.578960 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.578242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38082)
0620 19:00:45.578853 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.578906 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540}
W20260620 19:00:45.579000 9586 leader_election.cc:334] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.579052 9586 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e3933b420ab43a1841db9e0d591a974; no voters: c6fbd45b35e24825a1673ab2e1a17bd2, e8c6e1cbfd964be289e7b98e721c8c38
I20260620 19:00:45.579239 10306 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:00:45.579521 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:45.579281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38106)
0620 19:00:45.579437 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:45.579480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 19:00:45.579643 9769 leader_election.cc:334] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:45.579707 9769 leader_election.cc:302] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6fbd45b35e24825a1673ab2e1a17bd2; no voters: 4e3933b420ab43a1841db9e0d591a974, e8c6e1cbfd964be289e7b98e721c8c38
I20260620 19:00:45.579820 10307 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:00:46.060076 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:46.059657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:52903)
0620 19:00:46.059917 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:46.059989 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260620 19:00:46.064736 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:46.064338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:40383)
0620 19:00:46.064580 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:46.064642 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
I20260620 19:00:46.065423 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:46.065076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:60931)
0620 19:00:46.065295 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:46.065349 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 19:00:47.061036 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.060688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:59655)
0620 19:00:47.060891 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.060943 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260620 19:00:47.065771 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.065321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:52875)
0620 19:00:47.065611 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.065666 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":67,"threads_started":1}
I20260620 19:00:47.066349 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.066037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:38917)
0620 19:00:47.066228 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.066278 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":51,"threads_started":1}
I20260620 19:00:47.332623 10323 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:47.332748 10323 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:47.333149 10323 leader_election.cc:288] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189:39873)
I20260620 19:00:47.333598 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e3933b420ab43a1841db9e0d591a974" is_pre_election: true
I20260620 19:00:47.333734 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.333329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38324)
0620 19:00:47.333592 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.333644 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260620 19:00:47.333778 9611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6fbd45b35e24825a1673ab2e1a17bd2 in term 2.
I20260620 19:00:47.334167 9770 leader_election.cc:302] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e3933b420ab43a1841db9e0d591a974, c6fbd45b35e24825a1673ab2e1a17bd2; no voters:
I20260620 19:00:47.334334 10323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:00:47.334408 10323 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:47.334489 10323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:00:47.334656 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.334395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38340)
0620 19:00:47.334532 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.334590 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 19:00:47.334765 9769 leader_election.cc:334] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:47.335640 10323 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:47.335938 10323 leader_election.cc:288] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 election: Requested vote from peers e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415), 4e3933b420ab43a1841db9e0d591a974 (127.2.208.189:39873)
I20260620 19:00:47.336246 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.336023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38342)
0620 19:00:47.336132 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.336185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:00:47.336234 9611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e3933b420ab43a1841db9e0d591a974"
I20260620 19:00:47.336362 9611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:00:47.336835 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.336631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38350)
0620 19:00:47.336740 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.336782 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 19:00:47.336959 9769 leader_election.cc:334] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
I20260620 19:00:47.337414 9611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6fbd45b35e24825a1673ab2e1a17bd2 in term 3.
I20260620 19:00:47.337682 9770 leader_election.cc:302] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e3933b420ab43a1841db9e0d591a974, c6fbd45b35e24825a1673ab2e1a17bd2; no voters: e8c6e1cbfd964be289e7b98e721c8c38
I20260620 19:00:47.337858 10323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:00:47.338052 10323 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [term 3 LEADER]: Becoming Leader. State: Replica: c6fbd45b35e24825a1673ab2e1a17bd2, State: Running, Role: LEADER
I20260620 19:00:47.338308 10323 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } }
I20260620 19:00:47.339462 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6fbd45b35e24825a1673ab2e1a17bd2. Latest consensus state: current_term: 3 leader_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:47.339571 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:47.339833 10328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:47.341864 10328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:47.342247 10328 catalog_manager.cc:1259] Loaded cluster ID: 6b2e7e34ec74411887986c4e09133417
I20260620 19:00:47.342288 10328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:47.342813 10328 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:47.343174 10328 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Loaded TSK: 0
I20260620 19:00:47.343875 10328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:00:47.382560 9778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:47.385633 9778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:47.388403 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.388183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38354)
0620 19:00:47.388294 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.388348 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:00:47.388499 9611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [term 3 FOLLOWER]: Refusing update from remote peer c6fbd45b35e24825a1673ab2e1a17bd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 19:00:47.388937 10326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 19:00:47.389083 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.388829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.0.0.1:38364)
0620 19:00:47.388950 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.389029 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:00:47.389200 9769 consensus_peers.cc:487] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 -> Peer e8c6e1cbfd964be289e7b98e721c8c38 (127.2.208.190:39415): Couldn't send request to peer e8c6e1cbfd964be289e7b98e721c8c38. Status: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:00:47.390323 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6fbd45b35e24825a1673ab2e1a17bd2. Latest consensus state: current_term: 3 leader_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:47.390448 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:47.391481 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:47.391659 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:47.392675 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e8c6e1cbfd964be289e7b98e721c8c38" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39415 } } peers { permanent_uuid: "4e3933b420ab43a1841db9e0d591a974" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 39873 } attrs { promote: false } } peers { permanent_uuid: "c6fbd45b35e24825a1673ab2e1a17bd2" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33035 } attrs { promote: false } } }
I20260620 19:00:47.392782 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3933b420ab43a1841db9e0d591a974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:00:47.401888 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.397214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44091 (local address 127.0.0.1:35322)
0620 19:00:47.397434 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.397446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:47.397529 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:47.397953 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:47.397956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:47.397974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:47.398105 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.398112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399469 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.399473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.400601 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.400611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401455 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.401459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.401597 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.401620 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:00:47.401676 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:00:47.401742 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":74,"thread_start_us":83,"threads_started":1}
I20260620 19:00:47.402015 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.397741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.2.208.129:42729)
0620 19:00:47.397963 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.397974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.398243 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:00:47.398323 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.398328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.398385 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.398460 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:47.398681 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399509 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.400831 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401502 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.401596 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.401670 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.401803 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.401871 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20260620 19:00:47.402015 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.397090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.2.208.130:38639)
0620 19:00:47.397770 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.397774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.397788 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:47.397836 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.397839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.397892 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.398079 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:47.398394 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399064 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.400853 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401288 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.401399 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.401458 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.401819 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.401889 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":79,"threads_started":1}
I20260620 19:00:47.402117 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.396955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38639 (local address 127.0.0.1:51406)
0620 19:00:47.397063 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.397078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:47.397165 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:47.398015 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:47.398020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:47.398040 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:47.398271 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.398277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399149 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.399153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.400426 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.400436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401395 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.401399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.401515 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.401535 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:00:47.401636 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:00:47.401699 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":148}
I20260620 19:00:47.402247 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.397488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:42729 (local address 127.0.0.1:45370)
0620 19:00:47.398149 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.398163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:47.398250 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:47.398481 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:47.398484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:47.398499 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:47.398570 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.398575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399623 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.399626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.400752 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.400759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401603 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.401615 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.401695 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.401714 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:00:47.401781 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:00:47.401839 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":21,"thread_start_us":69,"threads_started":1}
I20260620 19:00:47.402617 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.397419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.2.208.131:44091)
0620 19:00:47.397716 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.397721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.397734 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:47.397781 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.397784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.397836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.398395 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:47.398507 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.399296 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.400755 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.401329 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.402316 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.402385 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.402422 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.402489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":90,"threads_started":1}
I20260620 19:00:47.402772 10034 tablet_service.cc:1467] Processing CreateTablet for tablet cf05084c463042d0b483fecf3193703c (DEFAULT_TABLE table=test-table [id=1818d22a0f5e462d9f50a325d7490906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:47.402849 9897 tablet_service.cc:1467] Processing CreateTablet for tablet cf05084c463042d0b483fecf3193703c (DEFAULT_TABLE table=test-table [id=1818d22a0f5e462d9f50a325d7490906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:47.403052 10034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05084c463042d0b483fecf3193703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:47.403143 9897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05084c463042d0b483fecf3193703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:47.405272 10340 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: Bootstrap starting.
I20260620 19:00:47.405431 10170 tablet_service.cc:1467] Processing CreateTablet for tablet cf05084c463042d0b483fecf3193703c (DEFAULT_TABLE table=test-table [id=1818d22a0f5e462d9f50a325d7490906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:47.405707 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05084c463042d0b483fecf3193703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:47.406078 10340 tablet_bootstrap.cc:654] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:47.406544 10341 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Bootstrap starting.
I20260620 19:00:47.407161 10340 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: No bootstrap required, opened a new log
I20260620 19:00:47.407266 10340 ts_tablet_manager.cc:1397] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:47.407291 10341 tablet_bootstrap.cc:654] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:47.407716 10340 raft_consensus.cc:348] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.407907 10340 raft_consensus.cc:374] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:47.407992 10340 raft_consensus.cc:729] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d662bdfa7e804fedace36555fe20d717, State: Initialized, Role: FOLLOWER
I20260620 19:00:47.408129 10340 consensus_queue.cc:260] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.408485 10340 ts_tablet_manager.cc:1428] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:47.408604 10341 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: No bootstrap required, opened a new log
I20260620 19:00:47.408685 10341 ts_tablet_manager.cc:1397] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:47.409041 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.408773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:40763)
0620 19:00:47.408923 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.408988 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 19:00:47.409117 10341 raft_consensus.cc:348] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.409228 10341 raft_consensus.cc:374] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:47.409269 10341 raft_consensus.cc:729] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bc808a023544749fd7aaf831414970, State: Initialized, Role: FOLLOWER
I20260620 19:00:47.409381 10341 consensus_queue.cc:260] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.409710 10341 ts_tablet_manager.cc:1428] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:47.410145 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.409787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:34275)
0620 19:00:47.410018 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.410070 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 19:00:47.410398 10344 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Bootstrap starting.
I20260620 19:00:47.410686 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.410399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:46303)
0620 19:00:47.410568 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.410621 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 19:00:47.411482 10344 tablet_bootstrap.cc:654] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:47.411749 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.411502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:44135)
0620 19:00:47.411643 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.411690 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 19:00:47.412518 10100 heartbeater.cc:503] Master 127.2.208.188:33035 was elected leader, sending a full tablet report...
I20260620 19:00:47.412608 9963 heartbeater.cc:503] Master 127.2.208.188:33035 was elected leader, sending a full tablet report...
I20260620 19:00:47.412673 10344 tablet_bootstrap.cc:492] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: No bootstrap required, opened a new log
I20260620 19:00:47.412744 10344 ts_tablet_manager.cc:1397] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:00:47.413179 10344 raft_consensus.cc:348] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.413293 10344 raft_consensus.cc:374] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:47.413347 10344 raft_consensus.cc:729] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e596d9f6904443e19f9dca54132ec024, State: Initialized, Role: FOLLOWER
I20260620 19:00:47.413461 10344 consensus_queue.cc:260] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.413846 10344 ts_tablet_manager.cc:1428] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:47.414506 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.414228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:43053)
0620 19:00:47.414393 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.414441 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 19:00:47.417409 10239 tablet.cc:2369] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:47.427537 10348 raft_consensus.cc:482] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:47.427642 10348 raft_consensus.cc:504] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.428560 10348 leader_election.cc:288] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639), d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
I20260620 19:00:47.433472 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.428760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:42729 (local address 127.2.208.131:45871)
0620 19:00:47.428933 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.428949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:47.429077 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:47.429558 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:47.429561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:47.429574 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:47.429719 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.429728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.430641 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.430644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.432201 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.432212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.433050 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.433054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.433154 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.433196 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:00:47.433270 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:47.433346 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":153}
I20260620 19:00:47.433472 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.428841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:45871 (local address 127.2.208.129:42729)
0620 19:00:47.428960 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.428982 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.429025 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:00:47.429180 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.429182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.429222 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.429286 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:47.429829 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.430544 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.432323 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.432912 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.433035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.433085 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.433296 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.433366 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:00:47.433704 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.428847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:51545 (local address 127.2.208.130:38639)
0620 19:00:47.428961 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.428986 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.429207 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:00:47.429341 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.429344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.429382 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.429461 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:47.429727 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.430520 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.432323 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.432928 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.433035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.433085 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.433443 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.433553 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:47.434005 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.428760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:38639 (local address 127.2.208.131:51545)
0620 19:00:47.429135 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.429148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:47.429233 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:47.429472 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:47.429475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:47.429492 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:47.429622 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.429629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.430641 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.430644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.432201 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:47.432212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.433152 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.433154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:47.433302 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.433325 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:00:47.433828 (+ 503us) client_negotiation.cc:240] Negotiation successful
0620 19:00:47.433906 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":176,"thread_start_us":111,"threads_started":1}
I20260620 19:00:47.434027 9917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05084c463042d0b483fecf3193703c" candidate_uuid: "e596d9f6904443e19f9dca54132ec024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d662bdfa7e804fedace36555fe20d717" is_pre_election: true
I20260620 19:00:47.434276 9917 raft_consensus.cc:2454] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e596d9f6904443e19f9dca54132ec024 in term 0.
I20260620 19:00:47.434504 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05084c463042d0b483fecf3193703c" candidate_uuid: "e596d9f6904443e19f9dca54132ec024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc808a023544749fd7aaf831414970" is_pre_election: true
I20260620 19:00:47.434648 10054 raft_consensus.cc:2454] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e596d9f6904443e19f9dca54132ec024 in term 0.
I20260620 19:00:47.434688 10123 leader_election.cc:302] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d662bdfa7e804fedace36555fe20d717, e596d9f6904443e19f9dca54132ec024; no voters:
I20260620 19:00:47.434969 10348 raft_consensus.cc:2781] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:47.435034 10348 raft_consensus.cc:482] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:47.435070 10348 raft_consensus.cc:3037] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:47.436092 10348 raft_consensus.cc:504] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.436457 10348 leader_election.cc:288] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [CANDIDATE]: Term 1 election: Requested vote from peers 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639), d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729)
I20260620 19:00:47.436725 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05084c463042d0b483fecf3193703c" candidate_uuid: "e596d9f6904443e19f9dca54132ec024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc808a023544749fd7aaf831414970"
I20260620 19:00:47.436813 9917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05084c463042d0b483fecf3193703c" candidate_uuid: "e596d9f6904443e19f9dca54132ec024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d662bdfa7e804fedace36555fe20d717"
I20260620 19:00:47.436856 10054 raft_consensus.cc:3037] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:47.436944 9917 raft_consensus.cc:3037] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:47.437729 10054 raft_consensus.cc:2454] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e596d9f6904443e19f9dca54132ec024 in term 1.
I20260620 19:00:47.437958 9917 raft_consensus.cc:2454] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e596d9f6904443e19f9dca54132ec024 in term 1.
I20260620 19:00:47.437987 10125 leader_election.cc:302] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42bc808a023544749fd7aaf831414970, e596d9f6904443e19f9dca54132ec024; no voters:
I20260620 19:00:47.438127 10348 raft_consensus.cc:2781] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:47.438359 10348 raft_consensus.cc:686] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 1 LEADER]: Becoming Leader. State: Replica: e596d9f6904443e19f9dca54132ec024, State: Running, Role: LEADER
I20260620 19:00:47.438553 10236 heartbeater.cc:503] Master 127.2.208.188:33035 was elected leader, sending a full tablet report...
I20260620 19:00:47.438635 10348 consensus_queue.cc:237] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } }
I20260620 19:00:47.439062 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.438805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:36875)
0620 19:00:47.438947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.439002 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 19:00:47.440918 9778 catalog_manager.cc:5495] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 reported cstate change: term changed from 0 to 1, leader changed from <none> to e596d9f6904443e19f9dca54132ec024 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "e596d9f6904443e19f9dca54132ec024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e596d9f6904443e19f9dca54132ec024" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:47.463678 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.460379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.2.208.131:44091)
0620 19:00:47.460495 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:47.460498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:47.460510 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:47.460635 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:47.460638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:47.460692 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:47.460778 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:47.460971 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.461463 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.462218 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:47.462449 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:47.462579 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:47.462847 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:47.463276 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:47.463414 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:47.463497 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:00:47.463548 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:00:47.465698 10103 tablet.cc:2369] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:47.466485 9917 raft_consensus.cc:1261] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Refusing update from remote peer e596d9f6904443e19f9dca54132ec024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:47.466965 10054 raft_consensus.cc:1261] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Refusing update from remote peer e596d9f6904443e19f9dca54132ec024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:47.467072 10350 consensus_queue.cc:1035] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d662bdfa7e804fedace36555fe20d717" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:47.467407 10348 consensus_queue.cc:1035] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42bc808a023544749fd7aaf831414970" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:47.469367 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.468750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:53467)
0620 19:00:47.469230 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.469294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260620 19:00:47.470403 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.470097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:46023)
0620 19:00:47.470286 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.470338 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 19:00:47.473038 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.472810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:35615)
0620 19:00:47.472913 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.472972 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:00:47.473838 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.473497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.131:48793)
0620 19:00:47.473723 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.473778 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 19:00:47.473955 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.473620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.129:49557)
0620 19:00:47.473840 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.473892 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 19:00:47.474771 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.474490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39415 (local address 127.2.208.130:54749)
0620 19:00:47.474659 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.474707 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:39415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 19:00:47.475858 10356 mvcc.cc:205] Tried to move back new op lower bound from 7298998466416967680 to 7298998466311192576. Current Snapshot: MvccSnapshot[applied={T|T < 7298998466416967680}]
I20260620 19:00:47.481873 9776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:47.482045 9776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:47.486354 9776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6fbd45b35e24825a1673ab2e1a17bd2: Sending DeleteTablet for 3 replicas of tablet cf05084c463042d0b483fecf3193703c
I20260620 19:00:47.487041 10034 tablet_service.cc:1514] Processing DeleteTablet for tablet cf05084c463042d0b483fecf3193703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:47 UTC) from {username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:51406
I20260620 19:00:47.487385 10364 tablet_replica.cc:330] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: stopping tablet replica
I20260620 19:00:47.487507 10364 raft_consensus.cc:2229] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:47.487636 10170 tablet_service.cc:1514] Processing DeleteTablet for tablet cf05084c463042d0b483fecf3193703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:47 UTC) from {username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:35322
I20260620 19:00:47.487679 10364 raft_consensus.cc:2258] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:47.488193 9897 tablet_service.cc:1514] Processing DeleteTablet for tablet cf05084c463042d0b483fecf3193703c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:47 UTC) from {username='oryx', principal='oryx/127.2.208.188@KRBTEST.COM'} at 127.0.0.1:45370
I20260620 19:00:47.488466 10364 ts_tablet_manager.cc:1905] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:47.488505 10366 tablet_replica.cc:330] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717: stopping tablet replica
I20260620 19:00:47.488616 10366 raft_consensus.cc:2229] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:47.488763 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9745
I20260620 19:00:47.488770 10366 raft_consensus.cc:2258] T cf05084c463042d0b483fecf3193703c P d662bdfa7e804fedace36555fe20d717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:47.491614 10364 ts_tablet_manager.cc:1918] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:47.491760 10364 log.cc:1196] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/cf05084c463042d0b483fecf3193703c
I20260620 19:00:47.492069 10364 ts_tablet_manager.cc:1939] T cf05084c463042d0b483fecf3193703c P 42bc808a023544749fd7aaf831414970: Deleting consensus metadata
I20260620 19:00:47.492820 9770 catalog_manager.cc:4841] TS 42bc808a023544749fd7aaf831414970 (127.2.208.130:38639): tablet cf05084c463042d0b483fecf3193703c (table test-table [id=1818d22a0f5e462d9f50a325d7490906]) successfully deleted
I20260620 19:00:47.496951 10365 tablet_replica.cc:330] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: stopping tablet replica
I20260620 19:00:47.497045 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9971
I20260620 19:00:47.497346 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:47.497015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:42729 (local address 127.0.0.1:45374)
0620 19:00:47.497199 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:47.497257 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:00:47.497090 10365 raft_consensus.cc:2229] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:47.498265 10365 raft_consensus.cc:2258] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:47.498967 10365 ts_tablet_manager.cc:1905] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:00:47.498946 9768 catalog_manager.cc:4584] TS d662bdfa7e804fedace36555fe20d717 (127.2.208.129:42729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf05084c463042d0b483fecf3193703c: Network error: Client connection negotiation failed: client connection to 127.2.208.129:42729: connect: Connection refused (error 111)
I20260620 19:00:47.501600 10365 ts_tablet_manager.cc:1918] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:47.501686 10365 log.cc:1196] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/cf05084c463042d0b483fecf3193703c
I20260620 19:00:47.502017 10365 ts_tablet_manager.cc:1939] T cf05084c463042d0b483fecf3193703c P e596d9f6904443e19f9dca54132ec024: Deleting consensus metadata
I20260620 19:00:47.502684 9768 catalog_manager.cc:4841] TS e596d9f6904443e19f9dca54132ec024 (127.2.208.131:44091): tablet cf05084c463042d0b483fecf3193703c (table test-table [id=1818d22a0f5e462d9f50a325d7490906]) successfully deleted
I20260620 19:00:47.503829 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 10107
I20260620 19:00:47.509524 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9465
I20260620 19:00:47.515136 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 9572
2026-06-20T19:00:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:47 dist-test-slave-tsm4 krb5kdc[10369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:47 dist-test-slave-tsm4 krb5kdc[10369](info): set up 2 sockets
Jun 20 19:00:47 dist-test-slave-tsm4 krb5kdc[10369](info): commencing operation
krb5kdc: starting...
W20260620 19:00:48.556023 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:48 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab.
Entry for principal oryx/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab.
I20260620 19:00:48.672266 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39329
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab
--principal=oryx/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:48.780333 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:48.780575 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:48.780614 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:48.784585 10382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:48.784637 10382 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:00:48.784654 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:48.784703 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:48.784729 10382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:48.784757 10382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:48.789968 10382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.190.keytab
--principal=oryx/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:48.790997 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:48.792079 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:48.800127 10390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:48.800158 10387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:48.800312 10382 server_base.cc:1034] running on GCE node
W20260620 19:00:48.800139 10388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:48.800967 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:48.801995 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:48.803179 10382 hybrid_clock.cc:648] HybridClock initialized: now 1781982048803159 us; error 31 us; skew 500 ppm
Jun 20 19:00:48 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982048, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:48.805707 10382 init.cc:377] Logged in from keytab as oryx/127.2.208.190@KRBTEST.COM (short username oryx)
I20260620 19:00:48.806510 10382 webserver.cc:460] Webserver started at http://127.2.208.190:43359/ using document root <none> and password file <none>
I20260620 19:00:48.807044 10382 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:48.807139 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:48.807399 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:48.809345 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "2c92a5cd859e4423ba9c4583af3e5174"
format_stamp: "Formatted at 2026-06-20 19:00:48 on dist-test-slave-tsm4"
server_key: "64c17819e28f3945859d6aa8847eabfc"
server_key_iv: "22c7f5435a672c4678ea57b994d882e8"
server_key_version: "encryptionkey@0"
I20260620 19:00:48.809888 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "2c92a5cd859e4423ba9c4583af3e5174"
format_stamp: "Formatted at 2026-06-20 19:00:48 on dist-test-slave-tsm4"
server_key: "64c17819e28f3945859d6aa8847eabfc"
server_key_iv: "22c7f5435a672c4678ea57b994d882e8"
server_key_version: "encryptionkey@0"
I20260620 19:00:48.813410 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:48.815761 10397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:48.816969 10382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:48.817103 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2c92a5cd859e4423ba9c4583af3e5174"
format_stamp: "Formatted at 2026-06-20 19:00:48 on dist-test-slave-tsm4"
server_key: "64c17819e28f3945859d6aa8847eabfc"
server_key_iv: "22c7f5435a672c4678ea57b994d882e8"
server_key_version: "encryptionkey@0"
I20260620 19:00:48.817211 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:48.834784 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:48.837522 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:48.837729 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:48.845314 10382 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39329
I20260620 19:00:48.845340 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39329 every 8 connection(s)
I20260620 19:00:48.846200 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:48.847692 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 10382
I20260620 19:00:48.847815 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:48.848085 2882 external_mini_cluster.cc:1428] Setting key 4eeb5233c8a5136fafb74082ae5481d6
I20260620 19:00:48.849002 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:48.854447 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Bootstrap starting.
Jun 20 19:00:48 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:48.856881 10460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:48.857594 10460 log.cc:824] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:48.859544 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: No bootstrap required, opened a new log
I20260620 19:00:48.862279 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:48.849424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.2.208.190:39329)
0620 19:00:48.849869 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:48.849888 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:48.849936 (+ 48us) server_negotiation.cc:407] Connection header received
0620 19:00:48.850545 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:48.850559 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:48.850857 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:48.851182 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:48.851986 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:48.852945 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:48.853767 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:48.854228 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:48.854387 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:48.857119 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:48.857123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:48.857135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:48.857160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:48.859609 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:48.860044 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:48.860050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:48.860056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:48.860168 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:48.860454 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:48.860457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:48.860459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:48.860837 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:48.861039 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:48.861306 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:00:48.861582 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
I20260620 19:00:48.862491 10460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } }
I20260620 19:00:48.862721 10460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:48.862797 10460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c92a5cd859e4423ba9c4583af3e5174, State: Initialized, Role: FOLLOWER
I20260620 19:00:48.863235 10460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } }
I20260620 19:00:48.863404 10460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:48.863488 10460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:48.863606 10460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:48.864475 10460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } }
I20260620 19:00:48.864779 10460 leader_election.cc:302] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c92a5cd859e4423ba9c4583af3e5174; no voters:
I20260620 19:00:48.865010 10460 leader_election.cc:288] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:48.865115 10465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:48.865339 10465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [term 1 LEADER]: Becoming Leader. State: Replica: 2c92a5cd859e4423ba9c4583af3e5174, State: Running, Role: LEADER
I20260620 19:00:48.865691 10465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } }
I20260620 19:00:48.866341 10460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:48.867442 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c92a5cd859e4423ba9c4583af3e5174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } } }
I20260620 19:00:48.867563 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:48.867924 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c92a5cd859e4423ba9c4583af3e5174. Latest consensus state: current_term: 1 leader_uuid: "2c92a5cd859e4423ba9c4583af3e5174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c92a5cd859e4423ba9c4583af3e5174" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39329 } } }
I20260620 19:00:48.868068 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174 [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:48.867980 10474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:48.870631 10474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:48.875629 10474 catalog_manager.cc:1347] Generated new cluster ID: ace01c2d810d421c86caf374980fe8a0
I20260620 19:00:48.875707 10474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:48.901657 10474 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:48.902623 10474 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:48.907215 10474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Generated new TSK 0
I20260620 19:00:48.907897 10474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
I20260620 19:00:48.936359 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39329
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:49.045083 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:49.045360 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:49.045434 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:49.049373 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:49.049456 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:49.049525 10486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:49.049602 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:49.054733 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:49.055768 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:49.057021 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:00:49.066293 10486 server_base.cc:1034] running on GCE node
W20260620 19:00:49.066254 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.066254 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.066255 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:49.066907 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:49.067504 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:49.068660 10486 hybrid_clock.cc:648] HybridClock initialized: now 1781982049068630 us; error 40 us; skew 500 ppm
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:49.071326 10486 init.cc:377] Logged in from keytab as oryx/127.2.208.129@KRBTEST.COM (short username oryx)
I20260620 19:00:49.072160 10486 webserver.cc:460] Webserver started at http://127.2.208.129:39297/ using document root <none> and password file <none>
I20260620 19:00:49.072645 10486 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:49.072702 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:49.072860 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:49.074753 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "42db8fe454ea40bd9808a7c0591d5faa"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "70e41c9a33e4cddef5934657b0de7468"
server_key_iv: "d0e0e3b919b88842d3571bf9f2cf26c2"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.075246 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "42db8fe454ea40bd9808a7c0591d5faa"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "70e41c9a33e4cddef5934657b0de7468"
server_key_iv: "d0e0e3b919b88842d3571bf9f2cf26c2"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.078641 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:49.080835 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.081974 10486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:49.082103 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "42db8fe454ea40bd9808a7c0591d5faa"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "70e41c9a33e4cddef5934657b0de7468"
server_key_iv: "d0e0e3b919b88842d3571bf9f2cf26c2"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.082213 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:49.091353 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:49.094380 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:49.094589 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:49.096067 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:49.096138 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.096223 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:49.096278 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:49.106671 10486 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:36649
I20260620 19:00:49.106720 10616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:36649 every 8 connection(s)
I20260620 19:00:49.107822 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:49.109670 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.096680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:50541 (local address 127.2.208.190:39329)
0620 19:00:49.096842 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.096847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.097658 (+ 811us) server_negotiation.cc:407] Connection header received
0620 19:00:49.098662 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.098666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.098719 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.098801 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.100338 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.101088 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.102203 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.102542 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.102659 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.105747 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.105751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.105753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.105779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.107580 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.108053 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.108057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.108059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.108151 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.108619 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.108622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.108624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.108826 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.108916 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.109364 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.109480 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:00:49.110384 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.097061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.129:50541)
0620 19:00:49.097488 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.097535 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.098463 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.099002 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.099013 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.099535 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.100196 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.100218 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.101206 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.101210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.102071 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.102080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.102705 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.102709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.102912 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.103519 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.103537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.105551 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.107677 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.107681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.107691 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.107978 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.108294 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.108297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.108299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.108500 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.108914 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.108920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.109154 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.109307 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.109557 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":124,"threads_started":1}
I20260620 19:00:49.111696 10617 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39329
I20260620 19:00:49.111850 10617 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:49.112368 10617 heartbeater.cc:511] Master 127.2.208.190:39329 requested a full tablet report, sending...
I20260620 19:00:49.112761 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 10486
I20260620 19:00:49.112891 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:49.113202 2882 external_mini_cluster.cc:1428] Setting key 5ace36b019cee7f4dfb96c7d9af45e42
I20260620 19:00:49.113940 10413 ts_manager.cc:194] Registered new tserver with Master: 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649)
I20260620 19:00:49.115284 10413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:50541
I20260620 19:00:49.121455 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.113808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:46645 (local address 127.2.208.190:39329)
0620 19:00:49.113910 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.113914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.114359 (+ 445us) server_negotiation.cc:407] Connection header received
0620 19:00:49.114464 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.114467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.114508 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.114586 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.115057 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.115781 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.116938 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.117327 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.117429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.118517 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.118520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.118531 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.118553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.119926 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.120347 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.120350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.120352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.120437 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.120700 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.120703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.120705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.120858 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.120954 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.121207 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.121311 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:00:49.121465 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.114006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.129:46645)
0620 19:00:49.114269 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.114283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.114374 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.114617 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.114620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.114809 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.114938 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.114944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.115953 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.115956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.116811 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.116822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.117442 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.117446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.117564 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.118049 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.118061 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.118356 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.120053 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.120056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.120057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.120243 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.120521 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.120525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.120526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.120608 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.120945 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.120948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.121042 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.121159 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.121286 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
I20260620 19:00:49.138242 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39329
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:49.246321 10626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:49.246554 10626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:49.246594 10626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:49.250384 10626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:49.250430 10626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:49.250466 10626 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:49.250497 10626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:49.255391 10626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:49.256269 10626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:49.257162 10626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:49.266305 10632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.266319 10631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.266316 10634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:49.266502 10626 server_base.cc:1034] running on GCE node
I20260620 19:00:49.267061 10626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:49.267726 10626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:49.268893 10626 hybrid_clock.cc:648] HybridClock initialized: now 1781982049268866 us; error 46 us; skew 500 ppm
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:49.271581 10626 init.cc:377] Logged in from keytab as oryx/127.2.208.130@KRBTEST.COM (short username oryx)
I20260620 19:00:49.272361 10626 webserver.cc:460] Webserver started at http://127.2.208.130:44185/ using document root <none> and password file <none>
I20260620 19:00:49.272917 10626 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:49.273006 10626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:49.273240 10626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:49.275213 10626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "ce83830b1a8c4f788ca30023534bbd74"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "68653f051614b628c3a962681c34bafb"
server_key_iv: "4a7761a4c6517e6720c4275d89834dc7"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.275812 10626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "ce83830b1a8c4f788ca30023534bbd74"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "68653f051614b628c3a962681c34bafb"
server_key_iv: "4a7761a4c6517e6720c4275d89834dc7"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.279313 10626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:00:49.281598 10641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.282768 10626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:00:49.282900 10626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "ce83830b1a8c4f788ca30023534bbd74"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "68653f051614b628c3a962681c34bafb"
server_key_iv: "4a7761a4c6517e6720c4275d89834dc7"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.283010 10626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:49.309183 10626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:49.312677 10626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:49.312873 10626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:49.314435 10626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:49.314510 10626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.314574 10626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:49.314607 10626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.325564 10626 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:43191
I20260620 19:00:49.325731 10756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:43191 every 8 connection(s)
I20260620 19:00:49.326563 10626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:49.331302 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.315392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:39493 (local address 127.2.208.190:39329)
0620 19:00:49.315542 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.315548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.319182 (+ 3634us) server_negotiation.cc:407] Connection header received
0620 19:00:49.320772 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.320777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.320837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.320891 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.322640 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.323308 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.325010 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.325326 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.325538 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.328181 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.328186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.328193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.328215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.329349 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.329830 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.329834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.329836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.329913 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.330270 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.330275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.330276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.330455 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.330561 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.331014 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.331126 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:00:49.332032 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.316014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.130:39493)
0620 19:00:49.318994 (+ 2980us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.319051 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.320507 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.321328 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.321337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.321878 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.322483 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.322497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.323507 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.323510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.324894 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.324902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.325721 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.325725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.325931 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.326465 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.326481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.328018 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.329461 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.329467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.329480 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.329728 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.330004 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.330008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.330011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.330146 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.330559 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.330565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.330795 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.330976 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.331262 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2744,"thread_start_us":137,"threads_started":1}
I20260620 19:00:49.333174 10757 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39329
I20260620 19:00:49.333331 10757 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:49.333793 10757 heartbeater.cc:511] Master 127.2.208.190:39329 requested a full tablet report, sending...
I20260620 19:00:49.334800 10414 ts_manager.cc:194] Registered new tserver with Master: ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191)
I20260620 19:00:49.335424 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 10626
I20260620 19:00:49.335561 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:49.335606 10414 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:39493
I20260620 19:00:49.335933 2882 external_mini_cluster.cc:1428] Setting key 424f152f3c3e9c02e9834842361e90d1
I20260620 19:00:49.342722 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.335234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:56785 (local address 127.2.208.190:39329)
0620 19:00:49.335328 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.335331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.335572 (+ 241us) server_negotiation.cc:407] Connection header received
0620 19:00:49.335735 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.335738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.335775 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.335851 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.336525 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.337099 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.338310 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.338634 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.338736 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.339884 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.339888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.339890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.339911 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.341171 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.341575 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.341580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.341581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.341660 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.341931 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.341936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.341937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.342141 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.342251 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.342483 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.342579 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:00:49.342742 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.335168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.130:56785)
0620 19:00:49.335495 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.335513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.335612 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.335996 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.336000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.336206 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.336386 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.336397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.337273 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.337276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.338189 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.338202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.338750 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.338756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.338880 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.339407 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.339426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.339772 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.341285 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.341288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.341290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.341475 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.341758 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.341761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.341763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.341845 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.342241 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.342244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.342334 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.342436 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.342567 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
I20260620 19:00:49.361838 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39329
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:49.470945 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:49.471215 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:49.471258 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:49.475262 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:49.475318 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:49.475410 10766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:00:49.475466 10766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:49.480749 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.208.190:39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:49.481745 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:49.482744 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:49.492115 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.492102 10774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:49.492136 10771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:49.492427 10766 server_base.cc:1034] running on GCE node
I20260620 19:00:49.492771 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:49.493345 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:49.494498 10766 hybrid_clock.cc:648] HybridClock initialized: now 1781982049494485 us; error 26 us; skew 500 ppm
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:49.497227 10766 init.cc:377] Logged in from keytab as oryx/127.2.208.131@KRBTEST.COM (short username oryx)
I20260620 19:00:49.497979 10766 webserver.cc:460] Webserver started at http://127.2.208.131:41289/ using document root <none> and password file <none>
I20260620 19:00:49.498479 10766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:49.498533 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:49.498688 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:49.500669 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "2f98425cbf32488886b70bf611392b7b"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "a9de1d013363b8ba7092821b2762614e"
server_key_iv: "ce9e46acce18e1ce0acc772acaf9f462"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.501165 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "2f98425cbf32488886b70bf611392b7b"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "a9de1d013363b8ba7092821b2762614e"
server_key_iv: "ce9e46acce18e1ce0acc772acaf9f462"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.504616 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:00:49.506922 10781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.508172 10766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:49.508301 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "2f98425cbf32488886b70bf611392b7b"
format_stamp: "Formatted at 2026-06-20 19:00:49 on dist-test-slave-tsm4"
server_key: "a9de1d013363b8ba7092821b2762614e"
server_key_iv: "ce9e46acce18e1ce0acc772acaf9f462"
server_key_version: "encryptionkey@0"
I20260620 19:00:49.508409 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:49.535657 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:49.538635 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:49.538831 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:49.540299 10766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:49.540345 10766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:49.540409 10766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:49.540442 10766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:49.551213 10766 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:43783
I20260620 19:00:49.551281 10896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:43783 every 8 connection(s)
I20260620 19:00:49.552098 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:49.553732 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.541158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:58383 (local address 127.2.208.190:39329)
0620 19:00:49.541284 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.541287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.542316 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 19:00:49.543501 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.543505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.543572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.543633 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.544991 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.545567 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.546730 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.546946 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.547058 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.550004 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.550007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.550010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.550042 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.551605 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.552079 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.552085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.552088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.552171 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.552742 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.552748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.552752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.552969 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.553085 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.553515 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.553618 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:00:49.554462 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.541707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.131:58383)
0620 19:00:49.542177 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.542217 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.543227 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.543967 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.543974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.544353 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.544852 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.544864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.545694 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.545700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.546608 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.546616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.547112 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.547116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.547273 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.547981 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.548008 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.549849 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.551715 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.551720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.551734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.551981 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.552290 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.552294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.552296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.552480 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.553085 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.553089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.553294 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.553441 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.553661 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":145,"threads_started":1}
I20260620 19:00:49.555666 10897 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39329
I20260620 19:00:49.555789 10897 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:49.556308 10897 heartbeater.cc:511] Master 127.2.208.190:39329 requested a full tablet report, sending...
I20260620 19:00:49.557369 10413 ts_manager.cc:194] Registered new tserver with Master: 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:49.557922 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 10766
I20260620 19:00:49.558059 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:49.558192 10413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:58383
I20260620 19:00:49.558436 2882 external_mini_cluster.cc:1428] Setting key 83f4372b194992905ab8a8310d484b64
I20260620 19:00:49.560389 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:00:49.565120 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.557764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.2.208.131:56609)
0620 19:00:49.558083 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.558099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.558194 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.558434 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.558436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.558633 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:49.558949 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.558959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.559884 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.559888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.560562 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.560570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.561003 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.561006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.561090 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.561567 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:49.561577 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:49.561864 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:49.563724 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.563727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.563728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.563911 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.564189 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:49.564191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:49.564192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:49.564260 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:49.564681 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:49.564683 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:49.564749 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.564824 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.564982 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260620 19:00:49.565258 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.557789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:56609 (local address 127.2.208.190:39329)
0620 19:00:49.557884 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.557889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.558156 (+ 267us) server_negotiation.cc:407] Connection header received
0620 19:00:49.558284 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.558287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.558324 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.558406 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.559061 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.559768 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.560676 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.560899 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.560999 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.561988 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.561993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.561999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.562028 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.563610 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.564002 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.564008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.564013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.564098 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.564342 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.564348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.564353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.564587 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.564696 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.564911 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.565037 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:00:49.585351 10463 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:49.579706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.2.208.190:39329)
0620 19:00:49.579809 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.579812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.579822 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:00:49.579859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.579861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.579895 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.579954 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.580631 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.581129 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.581927 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.582122 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.582199 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.585189 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47122: BlockingRecv error: recv got EOF from 127.0.0.1:47122 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:00:49 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:49.594455 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.586800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.2.208.190:39329)
0620 19:00:49.586957 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.586961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.587108 (+ 147us) server_negotiation.cc:407] Connection header received
0620 19:00:49.587251 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.587254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.587300 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.587404 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:49.587963 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.588461 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.589225 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.589447 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.589549 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.591728 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:49.591732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:49.591734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:49.591757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:49.592881 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.593269 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.593274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.593276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.593352 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:49.593653 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:49.593659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:49.593664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:49.593835 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:49.593969 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.594194 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.594300 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:00:49.597481 10413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:49.600421 10413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:49.614421 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:36649 (local address 127.0.0.1:48218)
0620 19:00:49.608507 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.608560 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.608913 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.610285 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.610288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.610314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:49.610437 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.610444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611926 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.611930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.613083 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.613089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.613889 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.613893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.613994 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.614035 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.614140 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.614261 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":73,"spinlock_wait_cycles":4864,"thread_start_us":62,"threads_started":1}
I20260620 19:00:49.614454 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43783 (local address 127.0.0.1:50512)
0620 19:00:49.608768 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.608783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.608915 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.610164 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.610167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.610185 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:49.610289 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.610295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611531 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.611536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.613075 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.613085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.614135 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.614138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.614218 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.614239 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.614295 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.614346 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":101,"spinlock_wait_cycles":5248,"thread_start_us":100,"threads_started":1}
I20260620 19:00:49.614719 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.2.208.131:43783)
0620 19:00:49.609763 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.609773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.609790 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:49.609849 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.609853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.610000 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.610174 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:49.610402 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611411 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.613206 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.614004 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.614131 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.614277 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.614330 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.614416 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":130,"threads_started":1}
I20260620 19:00:49.614946 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.2.208.129:36649)
0620 19:00:49.609586 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.609707 (+ 121us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.609731 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:00:49.609809 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.609815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.610001 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.610149 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:49.610563 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611687 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.613227 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.613771 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.613891 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.614043 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.614720 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.614814 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":78,"threads_started":1}
I20260620 19:00:49.615658 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.0.0.1:58670)
0620 19:00:49.609386 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.609399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.609506 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.609996 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.610002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.610022 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:49.610147 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.610155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611640 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.611643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.613898 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.613909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.615265 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.615268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.615352 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.615400 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.615467 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.615532 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"mutex_wait_us":1992,"thread_start_us":98,"threads_started":1}
I20260620 19:00:49.616060 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.608392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.2.208.130:43191)
0620 19:00:49.609263 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.609275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.609508 (+ 233us) server_negotiation.cc:407] Connection header received
0620 19:00:49.609616 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.609623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.609831 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.609996 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:49.610276 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.611540 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.614074 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.614899 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.614994 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.615167 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.615829 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.615918 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":129,"threads_started":1}
I20260620 19:00:49.617580 10551 tablet_service.cc:1467] Processing CreateTablet for tablet ce81da8604554faa9bd3da4b552bca71 (DEFAULT_TABLE table=test-table [id=9ae7ea045b564e96adf1c58767fc32d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.617676 10831 tablet_service.cc:1467] Processing CreateTablet for tablet ce81da8604554faa9bd3da4b552bca71 (DEFAULT_TABLE table=test-table [id=9ae7ea045b564e96adf1c58767fc32d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.618434 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce81da8604554faa9bd3da4b552bca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.618630 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce81da8604554faa9bd3da4b552bca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.618413 10688 tablet_service.cc:1467] Processing CreateTablet for tablet ce81da8604554faa9bd3da4b552bca71 (DEFAULT_TABLE table=test-table [id=9ae7ea045b564e96adf1c58767fc32d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.619262 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce81da8604554faa9bd3da4b552bca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.624686 10925 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Bootstrap starting.
I20260620 19:00:49.626367 10927 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Bootstrap starting.
I20260620 19:00:49.627157 10926 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Bootstrap starting.
I20260620 19:00:49.627172 10925 tablet_bootstrap.cc:654] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.628453 10927 tablet_bootstrap.cc:654] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.628422 10925 log.cc:824] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:49.628811 10926 tablet_bootstrap.cc:654] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.629262 10927 log.cc:824] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:49.629608 10926 log.cc:824] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:49.630689 10925 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: No bootstrap required, opened a new log
I20260620 19:00:49.630834 10927 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: No bootstrap required, opened a new log
I20260620 19:00:49.630884 10925 ts_tablet_manager.cc:1397] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:00:49.631026 10927 ts_tablet_manager.cc:1397] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:00:49.631309 10926 tablet_bootstrap.cc:492] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: No bootstrap required, opened a new log
I20260620 19:00:49.631501 10926 ts_tablet_manager.cc:1397] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:00:49.634018 10927 raft_consensus.cc:348] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.634243 10927 raft_consensus.cc:374] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.634300 10927 raft_consensus.cc:729] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.634750 10927 consensus_queue.cc:260] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.634738 10925 raft_consensus.cc:348] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.634999 10925 raft_consensus.cc:374] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.635061 10925 raft_consensus.cc:729] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.635116 10926 raft_consensus.cc:348] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.635332 10926 raft_consensus.cc:374] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.635406 10926 raft_consensus.cc:729] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.635494 10757 heartbeater.cc:503] Master 127.2.208.190:39329 was elected leader, sending a full tablet report...
I20260620 19:00:49.635519 10925 consensus_queue.cc:260] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.635774 10926 consensus_queue.cc:260] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.636386 10925 ts_tablet_manager.cc:1428] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:00:49.635742 10927 ts_tablet_manager.cc:1428] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:49.636562 10926 ts_tablet_manager.cc:1428] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:00:49.636715 10617 heartbeater.cc:503] Master 127.2.208.190:39329 was elected leader, sending a full tablet report...
I20260620 19:00:49.637106 10897 heartbeater.cc:503] Master 127.2.208.190:39329 was elected leader, sending a full tablet report...
I20260620 19:00:49.648063 10933 raft_consensus.cc:482] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:49.648204 10933 raft_consensus.cc:504] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.649134 10933 leader_election.cc:288] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191)
I20260620 19:00:49.652963 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.649503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:59915 (local address 127.2.208.130:43191)
0620 19:00:49.649637 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.649641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.649655 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:49.649706 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.649708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.649752 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.649837 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:49.650133 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.650752 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.651893 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.652476 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.652585 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.652623 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.652820 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.652871 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:00:49.652982 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.649332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.2.208.131:59915)
0620 19:00:49.649458 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.649474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.649559 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.649852 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.649854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.649889 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:49.650009 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.650015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.650856 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.650859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.651774 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.651782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.652582 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.652585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.652697 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.652722 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.652797 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.652861 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:00:49.653535 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce81da8604554faa9bd3da4b552bca71" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74" is_pre_election: true
I20260620 19:00:49.653816 10711 raft_consensus.cc:2454] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 0.
I20260620 19:00:49.654281 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.649333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:36649 (local address 127.2.208.131:52845)
0620 19:00:49.649805 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:49.649821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:49.649909 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:49.650154 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:49.650157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:49.650175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:49.650280 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.650286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.651008 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.651010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.652390 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:49.652400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.653246 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.653249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:49.653962 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.653981 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:00:49.654042 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:00:49.654133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":112,"thread_start_us":136,"threads_started":1}
I20260620 19:00:49.654301 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.649390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:52845 (local address 127.2.208.129:36649)
0620 19:00:49.649513 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.649518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.649898 (+ 380us) server_negotiation.cc:407] Connection header received
0620 19:00:49.650007 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.650010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.650059 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.650132 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:49.650382 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.650905 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.652508 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.653125 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.653241 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.653285 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.654079 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.654195 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:00:49.654317 10783 leader_election.cc:302] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f98425cbf32488886b70bf611392b7b, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:49.654666 10933 raft_consensus.cc:2781] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:49.654755 10933 raft_consensus.cc:482] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:49.654808 10933 raft_consensus.cc:3037] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:49.655179 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce81da8604554faa9bd3da4b552bca71" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa" is_pre_election: true
I20260620 19:00:49.655457 10571 raft_consensus.cc:2454] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 0.
I20260620 19:00:49.656601 10933 raft_consensus.cc:504] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.656965 10933 leader_election.cc:288] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 election: Requested vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191)
I20260620 19:00:49.657266 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce81da8604554faa9bd3da4b552bca71" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa"
I20260620 19:00:49.657274 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce81da8604554faa9bd3da4b552bca71" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74"
I20260620 19:00:49.657383 10711 raft_consensus.cc:3037] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:49.657383 10571 raft_consensus.cc:3037] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:49.658643 10571 raft_consensus.cc:2454] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 1.
I20260620 19:00:49.658651 10711 raft_consensus.cc:2454] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 1.
I20260620 19:00:49.658941 10782 leader_election.cc:302] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f98425cbf32488886b70bf611392b7b, 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:49.659154 10933 raft_consensus.cc:2781] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:49.659315 10933 raft_consensus.cc:686] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 1 LEADER]: Becoming Leader. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Running, Role: LEADER
I20260620 19:00:49.659638 10933 consensus_queue.cc:237] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.662668 10412 catalog_manager.cc:5495] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f98425cbf32488886b70bf611392b7b (127.2.208.131). New cstate: current_term: 1 leader_uuid: "2f98425cbf32488886b70bf611392b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } health_report { overall_health: HEALTHY } } }
I20260620 19:00:49.675458 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:49.671477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.2.208.131:43783)
0620 19:00:49.671591 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:49.671596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:49.671633 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:00:49.671826 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:49.671829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:49.671875 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:49.671975 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:49.672211 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.672702 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.673576 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:49.673882 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:49.673989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:49.674370 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:49.674902 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:49.675016 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:49.675168 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:00:49.675253 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:00:49.682674 10571 raft_consensus.cc:1261] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Refusing update from remote peer 2f98425cbf32488886b70bf611392b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:49.682832 10711 raft_consensus.cc:1261] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Refusing update from remote peer 2f98425cbf32488886b70bf611392b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:49.683475 10935 consensus_queue.cc:1035] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:49.683636 10933 consensus_queue.cc:1035] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:49.689939 10938 mvcc.cc:205] Tried to move back new op lower bound from 7298998475491971072 to 7298998475408502784. Current Snapshot: MvccSnapshot[applied={T|T < 7298998475491971072}]
I20260620 19:00:49.691416 10940 mvcc.cc:205] Tried to move back new op lower bound from 7298998475491971072 to 7298998475408502784. Current Snapshot: MvccSnapshot[applied={T|T < 7298998475491971072}]
I20260620 19:00:49.698716 10412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:49.698858 10412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:49.701176 10412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Sending DeleteTablet for 3 replicas of tablet ce81da8604554faa9bd3da4b552bca71
I20260620 19:00:49.701916 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet ce81da8604554faa9bd3da4b552bca71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:49 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:48218
I20260620 19:00:49.702066 10831 tablet_service.cc:1514] Processing DeleteTablet for tablet ce81da8604554faa9bd3da4b552bca71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:49 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 19:00:49.702466 10949 tablet_replica.cc:330] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: stopping tablet replica
I20260620 19:00:49.702446 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet ce81da8604554faa9bd3da4b552bca71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:49 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58670
I20260620 19:00:49.702795 10949 raft_consensus.cc:2229] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:49.702906 10951 tablet_replica.cc:330] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: stopping tablet replica
I20260620 19:00:49.703096 10949 raft_consensus.cc:2258] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:49.703269 10951 raft_consensus.cc:2229] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:49.703616 10951 raft_consensus.cc:2258] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:49.703677 10412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:49.704103 10412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:49.704501 10949 ts_tablet_manager.cc:1905] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:49.704917 10950 tablet_replica.cc:330] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: stopping tablet replica
I20260620 19:00:49.705181 10950 raft_consensus.cc:2229] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:49.705499 10950 raft_consensus.cc:2258] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:49.707154 10949 ts_tablet_manager.cc:1918] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:49.707257 10949 log.cc:1196] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/ce81da8604554faa9bd3da4b552bca71
I20260620 19:00:49.707563 10951 ts_tablet_manager.cc:1905] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:49.707585 10949 ts_tablet_manager.cc:1939] T ce81da8604554faa9bd3da4b552bca71 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting consensus metadata
I20260620 19:00:49.708752 10398 catalog_manager.cc:4841] TS 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649): tablet ce81da8604554faa9bd3da4b552bca71 (table test-table [id=9ae7ea045b564e96adf1c58767fc32d6]) successfully deleted
I20260620 19:00:49.709949 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c (DEFAULT_TABLE table=test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.709926 10831 tablet_service.cc:1467] Processing CreateTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c (DEFAULT_TABLE table=test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.710209 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e2c4107bd249a7b8e6bf1343a1097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.710296 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e2c4107bd249a7b8e6bf1343a1097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.710820 10951 ts_tablet_manager.cc:1918] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:49.710954 10951 log.cc:1196] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/ce81da8604554faa9bd3da4b552bca71
I20260620 19:00:49.711309 10951 ts_tablet_manager.cc:1939] T ce81da8604554faa9bd3da4b552bca71 P ce83830b1a8c4f788ca30023534bbd74: Deleting consensus metadata
I20260620 19:00:49.712785 10926 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Bootstrap starting.
I20260620 19:00:49.713315 10399 catalog_manager.cc:4841] TS ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191): tablet ce81da8604554faa9bd3da4b552bca71 (table test-table [id=9ae7ea045b564e96adf1c58767fc32d6]) successfully deleted
I20260620 19:00:49.713588 10688 tablet_service.cc:1467] Processing CreateTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c (DEFAULT_TABLE table=test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:49.713773 10926 tablet_bootstrap.cc:654] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.713810 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e2c4107bd249a7b8e6bf1343a1097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:49.714429 10925 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Bootstrap starting.
I20260620 19:00:49.715091 10926 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: No bootstrap required, opened a new log
I20260620 19:00:49.715171 10926 ts_tablet_manager.cc:1397] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:49.715478 10925 tablet_bootstrap.cc:654] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.715611 10926 raft_consensus.cc:348] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.715728 10926 raft_consensus.cc:374] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.715770 10926 raft_consensus.cc:729] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.715891 10926 consensus_queue.cc:260] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.716142 10926 ts_tablet_manager.cc:1428] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:49.716120 10950 ts_tablet_manager.cc:1905] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:49.716284 10927 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Bootstrap starting.
I20260620 19:00:49.717190 10927 tablet_bootstrap.cc:654] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:49.718719 10950 ts_tablet_manager.cc:1918] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:49.718809 10950 log.cc:1196] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/ce81da8604554faa9bd3da4b552bca71
I20260620 19:00:49.719130 10950 ts_tablet_manager.cc:1939] T ce81da8604554faa9bd3da4b552bca71 P 2f98425cbf32488886b70bf611392b7b: Deleting consensus metadata
I20260620 19:00:49.720034 10925 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: No bootstrap required, opened a new log
I20260620 19:00:49.720137 10925 ts_tablet_manager.cc:1397] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 19:00:49.720324 10400 catalog_manager.cc:4841] TS 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783): tablet ce81da8604554faa9bd3da4b552bca71 (table test-table [id=9ae7ea045b564e96adf1c58767fc32d6]) successfully deleted
I20260620 19:00:49.720705 10927 tablet_bootstrap.cc:492] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: No bootstrap required, opened a new log
I20260620 19:00:49.720819 10927 ts_tablet_manager.cc:1397] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:00:49.720953 10925 raft_consensus.cc:348] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.721066 10925 raft_consensus.cc:374] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.721102 10925 raft_consensus.cc:729] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.721203 10927 raft_consensus.cc:348] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.721251 10925 consensus_queue.cc:260] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.721318 10927 raft_consensus.cc:374] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:49.721361 10927 raft_consensus.cc:729] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Initialized, Role: FOLLOWER
I20260620 19:00:49.721529 10925 ts_tablet_manager.cc:1428] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 19:00:49.721508 10927 consensus_queue.cc:260] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:49.721750 10927 ts_tablet_manager.cc:1428] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 19:00:49.803000 10898 tablet.cc:2369] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:49.827754 10758 tablet.cc:2369] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:00:49.859601 10618 tablet.cc:2369] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:00:50.109563 10933 raft_consensus.cc:482] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:50.109709 10933 raft_consensus.cc:504] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.110162 10933 leader_election.cc:288] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191)
I20260620 19:00:50.110670 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e2c4107bd249a7b8e6bf1343a1097c" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74" is_pre_election: true
I20260620 19:00:50.110690 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e2c4107bd249a7b8e6bf1343a1097c" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa" is_pre_election: true
I20260620 19:00:50.110819 10571 raft_consensus.cc:2454] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 0.
I20260620 19:00:50.110832 10711 raft_consensus.cc:2454] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 0.
I20260620 19:00:50.111159 10783 leader_election.cc:302] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f98425cbf32488886b70bf611392b7b, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.111348 10933 raft_consensus.cc:2781] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:50.111428 10933 raft_consensus.cc:482] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:50.111495 10933 raft_consensus.cc:3037] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.112730 10933 raft_consensus.cc:504] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.113093 10933 leader_election.cc:288] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 election: Requested vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191)
I20260620 19:00:50.113392 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e2c4107bd249a7b8e6bf1343a1097c" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74"
I20260620 19:00:50.113390 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e2c4107bd249a7b8e6bf1343a1097c" candidate_uuid: "2f98425cbf32488886b70bf611392b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa"
I20260620 19:00:50.113526 10571 raft_consensus.cc:3037] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.113526 10711 raft_consensus.cc:3037] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.114383 10711 raft_consensus.cc:2454] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 1.
I20260620 19:00:50.114583 10571 raft_consensus.cc:2454] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f98425cbf32488886b70bf611392b7b in term 1.
I20260620 19:00:50.114872 10782 leader_election.cc:302] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f98425cbf32488886b70bf611392b7b, 42db8fe454ea40bd9808a7c0591d5faa; no voters:
I20260620 19:00:50.115106 10933 raft_consensus.cc:2781] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:50.115198 10933 raft_consensus.cc:686] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 1 LEADER]: Becoming Leader. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Running, Role: LEADER
I20260620 19:00:50.115346 10933 consensus_queue.cc:237] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.117116 10412 catalog_manager.cc:5495] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f98425cbf32488886b70bf611392b7b (127.2.208.131). New cstate: current_term: 1 leader_uuid: "2f98425cbf32488886b70bf611392b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } health_report { overall_health: HEALTHY } } }
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982048, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.190@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:50.193429 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.185289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.2.208.190:39329)
0620 19:00:50.185531 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.185537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.185552 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:00:50.185629 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.185631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.185677 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.185760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:50.186504 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.187177 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.187958 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.188173 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.188249 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.190373 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:50.190376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:50.190379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:50.190406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:50.191706 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:50.192193 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:50.192197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:50.192199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:50.192278 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:50.192548 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:50.192551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:50.192553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:50.192729 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:50.192827 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.193107 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.193265 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260620 19:00:50.193429 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.185184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39329 (local address 127.0.0.1:47146)
0620 19:00:50.185444 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.185460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.185563 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.185830 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.185832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.186253 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:50.186346 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.186351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.187268 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.187270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.187865 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.187873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.188267 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.188269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.188326 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.188883 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:50.188894 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:50.190247 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:50.191850 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:50.191858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:50.191865 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:50.192079 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:50.192377 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:50.192379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:50.192381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:50.192457 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:50.192836 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:50.192845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:50.192946 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.193055 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.193209 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260620 19:00:50.197216 10412 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47146:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:00:50.197976 10412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:50.202405 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 2508e873abab4b05888b735da3fc30ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b72758013d2a423c83f99a1a0e925653]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:50.202471 10831 tablet_service.cc:1467] Processing CreateTablet for tablet 2508e873abab4b05888b735da3fc30ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b72758013d2a423c83f99a1a0e925653]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:50.202420 10688 tablet_service.cc:1467] Processing CreateTablet for tablet 2508e873abab4b05888b735da3fc30ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b72758013d2a423c83f99a1a0e925653]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:00:50.202798 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2508e873abab4b05888b735da3fc30ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.202798 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2508e873abab4b05888b735da3fc30ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.202816 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2508e873abab4b05888b735da3fc30ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.205173 10927 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: Bootstrap starting.
I20260620 19:00:50.205399 10925 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: Bootstrap starting.
I20260620 19:00:50.205714 10926 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: Bootstrap starting.
I20260620 19:00:50.206111 10927 tablet_bootstrap.cc:654] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.206197 10925 tablet_bootstrap.cc:654] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.206408 10926 tablet_bootstrap.cc:654] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.207366 10925 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: No bootstrap required, opened a new log
I20260620 19:00:50.207441 10926 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: No bootstrap required, opened a new log
I20260620 19:00:50.207481 10925 ts_tablet_manager.cc:1397] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 19:00:50.207511 10926 ts_tablet_manager.cc:1397] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:50.207895 10925 raft_consensus.cc:348] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.207877 10926 raft_consensus.cc:348] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.208001 10925 raft_consensus.cc:374] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.208001 10926 raft_consensus.cc:374] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.208035 10925 raft_consensus.cc:729] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.208035 10926 raft_consensus.cc:729] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.208060 10927 tablet_bootstrap.cc:492] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: No bootstrap required, opened a new log
I20260620 19:00:50.208134 10927 ts_tablet_manager.cc:1397] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:50.208163 10926 consensus_queue.cc:260] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.208155 10925 consensus_queue.cc:260] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.208420 10926 ts_tablet_manager.cc:1428] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:50.208420 10925 ts_tablet_manager.cc:1428] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:00:50.208602 10927 raft_consensus.cc:348] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.208652 10933 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.208741 10927 raft_consensus.cc:374] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.208765 10933 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.208776 10927 raft_consensus.cc:729] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.208901 10927 consensus_queue.cc:260] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.208920 10932 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.209065 10932 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.209177 10927 ts_tablet_manager.cc:1428] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:50.209295 10931 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.209417 10931 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.229668 10931 raft_consensus.cc:482] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:50.229820 10931 raft_consensus.cc:504] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.230854 10931 leader_election.cc:288] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.236256 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.231081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:36649 (local address 127.2.208.130:36093)
0620 19:00:50.231432 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.231445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.231516 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.231788 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.231791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.231828 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:50.232132 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.232147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.233175 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.233178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.234765 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.234776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.235668 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.235673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.235841 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.235901 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.235975 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.236072 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":317,"spinlock_wait_cycles":28672,"thread_start_us":65,"threads_started":1}
I20260620 19:00:50.236353 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.231155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:36093 (local address 127.2.208.129:36649)
0620 19:00:50.231410 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.231414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.231513 (+ 99us) server_negotiation.cc:407] Connection header received
0620 19:00:50.231598 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.231601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.231648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.231760 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:50.232249 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.233039 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.234896 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.235542 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.235664 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.235718 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.236003 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.236219 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":52,"threads_started":1}
I20260620 19:00:50.236264 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.231069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43783 (local address 127.2.208.130:46401)
0620 19:00:50.231368 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.231404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.231506 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.231788 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.231791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.231829 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:50.232132 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.232140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.233175 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.233178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.234765 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.234776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.235676 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.235679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.235822 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.235894 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.235975 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.236072 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":322,"spinlock_wait_cycles":3456,"thread_start_us":63,"threads_started":1}
I20260620 19:00:50.236716 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.231136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:46401 (local address 127.2.208.131:43783)
0620 19:00:50.231559 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.231565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.231577 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:50.231619 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.231621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.231676 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.231771 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:50.232249 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.233039 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.234895 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.235540 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.235667 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.235716 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.236003 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.236075 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":58,"threads_started":1}
I20260620 19:00:50.236879 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2508e873abab4b05888b735da3fc30ab" candidate_uuid: "ce83830b1a8c4f788ca30023534bbd74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa" is_pre_election: true
I20260620 19:00:50.237030 10571 raft_consensus.cc:2454] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce83830b1a8c4f788ca30023534bbd74 in term 0.
I20260620 19:00:50.237430 10642 leader_election.cc:302] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.237561 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2508e873abab4b05888b735da3fc30ab" candidate_uuid: "ce83830b1a8c4f788ca30023534bbd74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b" is_pre_election: true
I20260620 19:00:50.237679 10931 raft_consensus.cc:2781] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:50.237737 10931 raft_consensus.cc:482] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:50.237782 10931 raft_consensus.cc:3037] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.237789 10851 raft_consensus.cc:2454] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce83830b1a8c4f788ca30023534bbd74 in term 0.
I20260620 19:00:50.238883 10931 raft_consensus.cc:504] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.239230 10931 leader_election.cc:288] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [CANDIDATE]: Term 1 election: Requested vote from peers 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.239528 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2508e873abab4b05888b735da3fc30ab" candidate_uuid: "ce83830b1a8c4f788ca30023534bbd74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b"
I20260620 19:00:50.239524 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2508e873abab4b05888b735da3fc30ab" candidate_uuid: "ce83830b1a8c4f788ca30023534bbd74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42db8fe454ea40bd9808a7c0591d5faa"
I20260620 19:00:50.239636 10851 raft_consensus.cc:3037] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.239636 10571 raft_consensus.cc:3037] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.240685 10571 raft_consensus.cc:2454] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce83830b1a8c4f788ca30023534bbd74 in term 1.
I20260620 19:00:50.240742 10851 raft_consensus.cc:2454] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce83830b1a8c4f788ca30023534bbd74 in term 1.
I20260620 19:00:50.240947 10642 leader_election.cc:302] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f98425cbf32488886b70bf611392b7b, 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.241086 10931 raft_consensus.cc:2781] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:50.241240 10931 raft_consensus.cc:686] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [term 1 LEADER]: Becoming Leader. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Running, Role: LEADER
I20260620 19:00:50.241474 10931 consensus_queue.cc:237] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.242969 10970 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: TxnStatusTablet state changed. Reason: New leader ce83830b1a8c4f788ca30023534bbd74. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.243072 10970 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:50.243312 10972 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:00:50.243484 10972 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:00:50.243528 10414 catalog_manager.cc:5495] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:50.264572 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.260857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.0.0.1:58676)
0620 19:00:50.260994 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.261009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.261105 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.261440 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.261444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.261450 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:50.261548 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.261554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.262289 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.262294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.262876 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.262884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.263313 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.263317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.263439 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.263486 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.264268 (+ 782us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.264282 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.264357 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.264441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:00:50.264591 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.260933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.2.208.130:43191)
0620 19:00:50.261181 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.261190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.261203 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:50.261268 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.261270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.261316 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.261432 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.261674 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.262187 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.262986 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.263212 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.263316 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.263620 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.264135 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.264283 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.264392 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.264440 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260620 19:00:50.267256 10571 raft_consensus.cc:1261] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Refusing update from remote peer ce83830b1a8c4f788ca30023534bbd74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.267455 10851 raft_consensus.cc:1261] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Refusing update from remote peer ce83830b1a8c4f788ca30023534bbd74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.267737 10931 consensus_queue.cc:1035] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.268004 10970 consensus_queue.cc:1035] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.269251 10932 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: TxnStatusTablet state changed. Reason: New leader ce83830b1a8c4f788ca30023534bbd74. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.269375 10932 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.269618 10933 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: TxnStatusTablet state changed. Reason: New leader ce83830b1a8c4f788ca30023534bbd74. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.269745 10933 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.272387 10931 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.272508 10931 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:50.272625 10931 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.272719 10931 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P ce83830b1a8c4f788ca30023534bbd74: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:00:50.273008 10933 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.273108 10933 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 2f98425cbf32488886b70bf611392b7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.274132 10932 tablet_replica.cc:439] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce83830b1a8c4f788ca30023534bbd74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } } }
I20260620 19:00:50.274242 10932 tablet_replica.cc:442] T 2508e873abab4b05888b735da3fc30ab P 42db8fe454ea40bd9808a7c0591d5faa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:00:50.286695 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.283112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:43887 (local address 127.2.208.130:43191)
0620 19:00:50.283233 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.283238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.283398 (+ 160us) server_negotiation.cc:407] Connection header received
0620 19:00:50.283520 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.283522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.283559 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.283619 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.283838 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.284318 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.285549 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.285823 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.285955 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.286235 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.286314 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.286396 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.286498 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.286541 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:00:50.286705 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.282995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.2.208.131:43887)
0620 19:00:50.283297 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.283311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.283433 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.283647 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.283650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.283658 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:50.283754 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.283760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.284405 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.284408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.285397 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.285407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.285922 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.285926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.286010 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.286058 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.286410 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.286416 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.286481 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.286560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260620 19:00:50.292124 10711 raft_consensus.cc:1261] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Refusing update from remote peer 2f98425cbf32488886b70bf611392b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.292223 10571 raft_consensus.cc:1261] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Refusing update from remote peer 2f98425cbf32488886b70bf611392b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.292520 10946 consensus_queue.cc:1035] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.292670 10933 consensus_queue.cc:1035] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.297154 10976 mvcc.cc:205] Tried to move back new op lower bound from 7298998477993197568 to 7298998477274431488. Current Snapshot: MvccSnapshot[applied={T|T < 7298998477993197568}]
I20260620 19:00:50.313648 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.310042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:60623 (local address 127.2.208.131:43783)
0620 19:00:50.310170 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.310175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.310380 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:00:50.310507 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.310510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.310547 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.310613 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.310857 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.311538 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.312541 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.312806 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.312936 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.313225 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.313301 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.313376 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.313458 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.313504 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:00:50.313663 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.309988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43783 (local address 127.2.208.130:60623)
0620 19:00:50.310305 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.310320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.310422 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.310636 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.310639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.310646 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:50.310764 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.310782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.311631 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.311637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.312382 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.312391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.312947 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.312950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.313043 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.313092 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.313380 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.313388 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.313450 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.313521 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
I20260620 19:00:50.339228 10413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.339401 10413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.341624 10413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Sending DeleteTablet for 3 replicas of tablet 61e2c4107bd249a7b8e6bf1343a1097c
I20260620 19:00:50.342097 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:48218
I20260620 19:00:50.342166 10831 tablet_service.cc:1514] Processing DeleteTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 19:00:50.342555 10994 tablet_replica.cc:330] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: stopping tablet replica
I20260620 19:00:50.342733 10995 tablet_replica.cc:330] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: stopping tablet replica
I20260620 19:00:50.342881 10995 raft_consensus.cc:2229] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:50.343034 10994 raft_consensus.cc:2229] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.343250 10995 raft_consensus.cc:2258] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.343425 10994 raft_consensus.cc:2258] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.344126 10995 ts_tablet_manager.cc:1905] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.345618 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet 61e2c4107bd249a7b8e6bf1343a1097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58670
I20260620 19:00:50.345939 10997 tablet_replica.cc:330] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: stopping tablet replica
I20260620 19:00:50.346087 10997 raft_consensus.cc:2229] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.346230 10994 ts_tablet_manager.cc:1905] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.346240 10997 raft_consensus.cc:2258] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.346832 10997 ts_tablet_manager.cc:1905] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.349319 10994 ts_tablet_manager.cc:1918] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:50.349398 10994 log.cc:1196] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/61e2c4107bd249a7b8e6bf1343a1097c
I20260620 19:00:50.349334 10997 ts_tablet_manager.cc:1918] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:50.349538 10997 log.cc:1196] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/61e2c4107bd249a7b8e6bf1343a1097c
I20260620 19:00:50.349778 10994 ts_tablet_manager.cc:1939] T 61e2c4107bd249a7b8e6bf1343a1097c P 42db8fe454ea40bd9808a7c0591d5faa: Deleting consensus metadata
I20260620 19:00:50.349903 10997 ts_tablet_manager.cc:1939] T 61e2c4107bd249a7b8e6bf1343a1097c P ce83830b1a8c4f788ca30023534bbd74: Deleting consensus metadata
I20260620 19:00:50.350478 10398 catalog_manager.cc:4841] TS 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649): tablet 61e2c4107bd249a7b8e6bf1343a1097c (table test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]) successfully deleted
I20260620 19:00:50.350523 10399 catalog_manager.cc:4841] TS ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191): tablet 61e2c4107bd249a7b8e6bf1343a1097c (table test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]) successfully deleted
I20260620 19:00:50.352115 10995 ts_tablet_manager.cc:1918] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:50.352195 10995 log.cc:1196] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/61e2c4107bd249a7b8e6bf1343a1097c
I20260620 19:00:50.352465 10995 ts_tablet_manager.cc:1939] T 61e2c4107bd249a7b8e6bf1343a1097c P 2f98425cbf32488886b70bf611392b7b: Deleting consensus metadata
I20260620 19:00:50.353144 10400 catalog_manager.cc:4841] TS 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783): tablet 61e2c4107bd249a7b8e6bf1343a1097c (table test-table [id=fb171bf5c3bb4a28b1c12dd0c766e673]) successfully deleted
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:00:50.379942 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:50.370805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.2.208.190:39329)
0620 19:00:50.371484 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.371489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.371503 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:50.371552 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.371556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.371615 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.371689 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:50.372633 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.373694 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.375116 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.375625 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.375736 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.379596 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47152: BlockingRecv error: recv got EOF from 127.0.0.1:47152 (error 108)
Metrics: {"server-negotiator.queue_time_us":520}
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[10369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM
I20260620 19:00:50.402462 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.391813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.2.208.190:39329)
0620 19:00:50.391939 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.391944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.392070 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:00:50.392265 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.392270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.392320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.392423 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:50.393268 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.394072 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.395317 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.395690 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.395793 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.398768 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:50.398772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:50.398774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:50.398803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:50.400555 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:50.401062 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:50.401066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:50.401068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:50.401157 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:50.401486 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:50.401490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:50.401491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:50.401712 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:50.401813 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.402181 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.402304 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:00:50.405580 10413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:50.406041 10413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:50.411218 10551 tablet_service.cc:1467] Processing CreateTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (DEFAULT_TABLE table=test-table [id=19c9927cbafa4de8a84fa90396607277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.411586 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02ca6d49f804296a1ce1bc9f39b4ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.411609 10831 tablet_service.cc:1467] Processing CreateTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (DEFAULT_TABLE table=test-table [id=19c9927cbafa4de8a84fa90396607277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.411927 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02ca6d49f804296a1ce1bc9f39b4ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.414011 10925 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Bootstrap starting.
I20260620 19:00:50.414413 10926 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Bootstrap starting.
I20260620 19:00:50.414649 10688 tablet_service.cc:1467] Processing CreateTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (DEFAULT_TABLE table=test-table [id=19c9927cbafa4de8a84fa90396607277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.414899 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a02ca6d49f804296a1ce1bc9f39b4ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.414986 10925 tablet_bootstrap.cc:654] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.415212 10926 tablet_bootstrap.cc:654] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.416538 10926 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: No bootstrap required, opened a new log
I20260620 19:00:50.416621 10926 ts_tablet_manager.cc:1397] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:50.417022 10927 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Bootstrap starting.
I20260620 19:00:50.417006 10926 raft_consensus.cc:348] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.417105 10926 raft_consensus.cc:374] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.417135 10926 raft_consensus.cc:729] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.417268 10926 consensus_queue.cc:260] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.417614 10926 ts_tablet_manager.cc:1428] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:50.417825 10927 tablet_bootstrap.cc:654] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.418968 10927 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: No bootstrap required, opened a new log
I20260620 19:00:50.419056 10927 ts_tablet_manager.cc:1397] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:50.419670 10927 raft_consensus.cc:348] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.419790 10925 tablet_bootstrap.cc:492] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: No bootstrap required, opened a new log
I20260620 19:00:50.419788 10927 raft_consensus.cc:374] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.419826 10927 raft_consensus.cc:729] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.419868 10925 ts_tablet_manager.cc:1397] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:00:50.419960 10927 consensus_queue.cc:260] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.420221 10927 ts_tablet_manager.cc:1428] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:00:50.420291 10925 raft_consensus.cc:348] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.420408 10925 raft_consensus.cc:374] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.420446 10925 raft_consensus.cc:729] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.420573 10925 consensus_queue.cc:260] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.420842 10925 ts_tablet_manager.cc:1428] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 19:00:50.641700 10932 raft_consensus.cc:482] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:50.641847 10932 raft_consensus.cc:504] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.643028 10932 leader_election.cc:288] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.648098 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.643339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:54769 (local address 127.2.208.130:43191)
0620 19:00:50.643493 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.643498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.643804 (+ 306us) server_negotiation.cc:407] Connection header received
0620 19:00:50.643974 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.643978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.644034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.644118 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:50.644507 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.645161 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.646697 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.647345 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.647501 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.647568 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.647900 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.647997 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:00:50.648099 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.643419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:39731 (local address 127.2.208.131:43783)
0620 19:00:50.643547 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.643553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.643875 (+ 322us) server_negotiation.cc:407] Connection header received
0620 19:00:50.643987 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.643990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.644034 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.644118 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:00:50.644844 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.645465 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.646840 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.647472 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.647571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.647624 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.647900 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.647971 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:00:50.648198 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.643248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.2.208.129:54769)
0620 19:00:50.643692 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.643708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.643837 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.644150 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.644153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.644191 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:50.644400 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.644409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.645266 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.645269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.646553 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.646563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.647530 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.647536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.647714 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.647790 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.647869 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.648000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":136,"spinlock_wait_cycles":24320,"thread_start_us":179,"threads_started":1}
I20260620 19:00:50.648198 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.643299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43783 (local address 127.2.208.129:39731)
0620 19:00:50.643791 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.643802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.643883 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.644150 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.644153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.644193 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:00:50.644732 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.644739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.645583 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.645586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.646715 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.646722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.647566 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.647569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.647728 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.647797 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.647869 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.648000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":161,"spinlock_wait_cycles":8320,"thread_start_us":134,"threads_started":1}
I20260620 19:00:50.648665 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02ca6d49f804296a1ce1bc9f39b4ca0" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74" is_pre_election: true
I20260620 19:00:50.648818 10711 raft_consensus.cc:2454] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 0.
I20260620 19:00:50.648846 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02ca6d49f804296a1ce1bc9f39b4ca0" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b" is_pre_election: true
I20260620 19:00:50.648998 10851 raft_consensus.cc:2454] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 0.
I20260620 19:00:50.649223 10503 leader_election.cc:302] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.649524 10932 raft_consensus.cc:2781] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:50.649585 10932 raft_consensus.cc:482] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:50.649623 10932 raft_consensus.cc:3037] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.650565 10932 raft_consensus.cc:504] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.650902 10932 leader_election.cc:288] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 election: Requested vote from peers ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.651191 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02ca6d49f804296a1ce1bc9f39b4ca0" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74"
I20260620 19:00:50.651305 10711 raft_consensus.cc:3037] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.651290 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02ca6d49f804296a1ce1bc9f39b4ca0" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b"
I20260620 19:00:50.651440 10851 raft_consensus.cc:3037] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.652222 10711 raft_consensus.cc:2454] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 1.
I20260620 19:00:50.652388 10851 raft_consensus.cc:2454] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 1.
I20260620 19:00:50.652477 10503 leader_election.cc:302] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.652664 10932 raft_consensus.cc:2781] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:50.652878 10932 raft_consensus.cc:686] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 LEADER]: Becoming Leader. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Running, Role: LEADER
I20260620 19:00:50.653131 10932 consensus_queue.cc:237] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.655303 10412 catalog_manager.cc:5495] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa reported cstate change: term changed from 0 to 1, leader changed from <none> to 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129). New cstate: current_term: 1 leader_uuid: "42db8fe454ea40bd9808a7c0591d5faa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:50.718712 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.715501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.2.208.129:36649)
0620 19:00:50.715633 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.715637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.715651 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:50.715773 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.715775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.715853 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.715948 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.716120 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.716611 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.717407 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.717626 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.717697 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.717907 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.718345 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.718427 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.718485 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.718542 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:00:50.721340 10711 raft_consensus.cc:1261] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Refusing update from remote peer 42db8fe454ea40bd9808a7c0591d5faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.721546 10851 raft_consensus.cc:1261] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Refusing update from remote peer 42db8fe454ea40bd9808a7c0591d5faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.721930 11014 consensus_queue.cc:1035] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.722129 10932 consensus_queue.cc:1035] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.732897 10413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.733065 10413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.735004 10413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Sending DeleteTablet for 3 replicas of tablet a02ca6d49f804296a1ce1bc9f39b4ca0
I20260620 19:00:50.735574 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:48218
I20260620 19:00:50.735587 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58670
I20260620 19:00:50.735775 10994 tablet_replica.cc:330] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: stopping tablet replica
I20260620 19:00:50.735859 10831 tablet_service.cc:1514] Processing DeleteTablet for tablet a02ca6d49f804296a1ce1bc9f39b4ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 19:00:50.735901 10994 raft_consensus.cc:2229] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:50.736160 10994 raft_consensus.cc:2258] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.736346 10997 tablet_replica.cc:330] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: stopping tablet replica
I20260620 19:00:50.736460 10997 raft_consensus.cc:2229] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.736596 10997 raft_consensus.cc:2258] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.736765 10995 tablet_replica.cc:330] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: stopping tablet replica
I20260620 19:00:50.736877 10995 raft_consensus.cc:2229] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.736999 10995 raft_consensus.cc:2258] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.736943 10413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:00:50.737354 10413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:00:50.737790 10997 ts_tablet_manager.cc:1905] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.737844 10994 ts_tablet_manager.cc:1905] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.739995 10997 ts_tablet_manager.cc:1918] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:50.740046 10997 log.cc:1196] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/a02ca6d49f804296a1ce1bc9f39b4ca0
I20260620 19:00:50.740236 10997 ts_tablet_manager.cc:1939] T a02ca6d49f804296a1ce1bc9f39b4ca0 P ce83830b1a8c4f788ca30023534bbd74: Deleting consensus metadata
I20260620 19:00:50.740432 10994 ts_tablet_manager.cc:1918] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:50.740499 10994 log.cc:1196] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/a02ca6d49f804296a1ce1bc9f39b4ca0
I20260620 19:00:50.740753 10994 ts_tablet_manager.cc:1939] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting consensus metadata
I20260620 19:00:50.740875 10399 catalog_manager.cc:4841] TS ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191): tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (table test-table [id=19c9927cbafa4de8a84fa90396607277]) successfully deleted
I20260620 19:00:50.741335 10398 catalog_manager.cc:4841] TS 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649): tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (table test-table [id=19c9927cbafa4de8a84fa90396607277]) successfully deleted
I20260620 19:00:50.742004 10995 ts_tablet_manager.cc:1905] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.743986 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 (DEFAULT_TABLE table=test-table [id=16ceb22ccb704246b51e8d581b8aa3c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.744247 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dfb0da172d47a8a4957fc8c9810261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.744232 10995 ts_tablet_manager.cc:1918] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:00:50.744295 10995 log.cc:1196] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/a02ca6d49f804296a1ce1bc9f39b4ca0
I20260620 19:00:50.744566 10995 ts_tablet_manager.cc:1939] T a02ca6d49f804296a1ce1bc9f39b4ca0 P 2f98425cbf32488886b70bf611392b7b: Deleting consensus metadata
I20260620 19:00:50.745146 10400 catalog_manager.cc:4841] TS 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783): tablet a02ca6d49f804296a1ce1bc9f39b4ca0 (table test-table [id=19c9927cbafa4de8a84fa90396607277]) successfully deleted
I20260620 19:00:50.746197 10925 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: Bootstrap starting.
I20260620 19:00:50.746390 10831 tablet_service.cc:1467] Processing CreateTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 (DEFAULT_TABLE table=test-table [id=16ceb22ccb704246b51e8d581b8aa3c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.746622 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dfb0da172d47a8a4957fc8c9810261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.747112 10925 tablet_bootstrap.cc:654] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.748510 10926 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Bootstrap starting.
I20260620 19:00:50.749285 10926 tablet_bootstrap.cc:654] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.749811 10925 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: No bootstrap required, opened a new log
I20260620 19:00:50.749889 10925 ts_tablet_manager.cc:1397] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:00:50.749868 10688 tablet_service.cc:1467] Processing CreateTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 (DEFAULT_TABLE table=test-table [id=16ceb22ccb704246b51e8d581b8aa3c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:00:50.750089 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dfb0da172d47a8a4957fc8c9810261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:50.750255 10925 raft_consensus.cc:348] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.750335 10925 raft_consensus.cc:374] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.750355 10925 raft_consensus.cc:729] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.750473 10925 consensus_queue.cc:260] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.750772 10925 ts_tablet_manager.cc:1428] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:50.751890 10926 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: No bootstrap required, opened a new log
I20260620 19:00:50.751964 10926 ts_tablet_manager.cc:1397] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:00:50.752290 10926 raft_consensus.cc:348] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.752380 10926 raft_consensus.cc:374] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.752408 10926 raft_consensus.cc:729] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f98425cbf32488886b70bf611392b7b, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.752532 10926 consensus_queue.cc:260] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.752792 10926 ts_tablet_manager.cc:1428] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:50.754741 10927 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Bootstrap starting.
I20260620 19:00:50.755602 10927 tablet_bootstrap.cc:654] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:50.756723 10927 tablet_bootstrap.cc:492] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: No bootstrap required, opened a new log
I20260620 19:00:50.756790 10927 ts_tablet_manager.cc:1397] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:00:50.757185 10927 raft_consensus.cc:348] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.757272 10927 raft_consensus.cc:374] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:50.757292 10927 raft_consensus.cc:729] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce83830b1a8c4f788ca30023534bbd74, State: Initialized, Role: FOLLOWER
I20260620 19:00:50.757424 10927 consensus_queue.cc:260] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.757668 10927 ts_tablet_manager.cc:1428] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:00:50.827932 10932 raft_consensus.cc:482] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:00:50.828063 10932 raft_consensus.cc:504] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.828567 10932 leader_election.cc:288] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.828979 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dfb0da172d47a8a4957fc8c9810261" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b" is_pre_election: true
I20260620 19:00:50.828960 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dfb0da172d47a8a4957fc8c9810261" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74" is_pre_election: true
I20260620 19:00:50.829105 10711 raft_consensus.cc:2454] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 0.
I20260620 19:00:50.829105 10851 raft_consensus.cc:2454] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 0.
I20260620 19:00:50.829427 10503 leader_election.cc:302] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.829648 10932 raft_consensus.cc:2781] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:00:50.829710 10932 raft_consensus.cc:482] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:00:50.829761 10932 raft_consensus.cc:3037] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.830720 10932 raft_consensus.cc:504] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.831074 10932 leader_election.cc:288] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 election: Requested vote from peers ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191), 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783)
I20260620 19:00:50.831357 10711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dfb0da172d47a8a4957fc8c9810261" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce83830b1a8c4f788ca30023534bbd74"
I20260620 19:00:50.831514 10711 raft_consensus.cc:3037] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.831459 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dfb0da172d47a8a4957fc8c9810261" candidate_uuid: "42db8fe454ea40bd9808a7c0591d5faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f98425cbf32488886b70bf611392b7b"
I20260620 19:00:50.831606 10851 raft_consensus.cc:3037] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:50.832406 10711 raft_consensus.cc:2454] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 1.
I20260620 19:00:50.832685 10503 leader_election.cc:302] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db8fe454ea40bd9808a7c0591d5faa, ce83830b1a8c4f788ca30023534bbd74; no voters:
I20260620 19:00:50.832770 10851 raft_consensus.cc:2454] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db8fe454ea40bd9808a7c0591d5faa in term 1.
I20260620 19:00:50.832875 10932 raft_consensus.cc:2781] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:00:50.832978 10932 raft_consensus.cc:686] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 LEADER]: Becoming Leader. State: Replica: 42db8fe454ea40bd9808a7c0591d5faa, State: Running, Role: LEADER
I20260620 19:00:50.833108 10932 consensus_queue.cc:237] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } }
I20260620 19:00:50.834718 10413 catalog_manager.cc:5495] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa reported cstate change: term changed from 0 to 1, leader changed from <none> to 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129). New cstate: current_term: 1 leader_uuid: "42db8fe454ea40bd9808a7c0591d5faa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42db8fe454ea40bd9808a7c0591d5faa" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 36649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260620 19:00:50.865012 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.861049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:47585 (local address 127.2.208.130:43191)
0620 19:00:50.861177 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.861182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.861371 (+ 189us) server_negotiation.cc:407] Connection header received
0620 19:00:50.861539 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.861542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.861594 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.861669 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.862006 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.862563 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.863773 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.864074 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.864202 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.864561 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.864643 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.864721 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.864825 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.864874 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:00:50.865098 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.860996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43191 (local address 127.2.208.129:47585)
0620 19:00:50.861275 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.861298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.861418 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.861790 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.861792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.861799 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:50.861894 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.861898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.862681 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.862685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.863649 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.863658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.864216 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.864219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.864380 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.864432 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.864723 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.864731 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.864799 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.864875 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260620 19:00:50.870543 10711 raft_consensus.cc:1261] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Refusing update from remote peer 42db8fe454ea40bd9808a7c0591d5faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.870685 10850 raft_consensus.cc:1261] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Refusing update from remote peer 42db8fe454ea40bd9808a7c0591d5faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:00:50.871265 11019 consensus_queue.cc:1035] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce83830b1a8c4f788ca30023534bbd74" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.871464 10932 consensus_queue.cc:1035] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f98425cbf32488886b70bf611392b7b" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:00:50.890755 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.886843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:36649 (local address 127.2.208.130:53755)
0620 19:00:50.887105 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.887125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:50.887256 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:50.887531 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:50.887534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:50.887542 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:00:50.887653 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.887665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.888531 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.888537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.889411 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:50.889421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.889855 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.889858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:50.889961 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.889986 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.890458 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.890464 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:00:50.890541 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:00:50.890618 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":112,"threads_started":1}
I20260620 19:00:50.890751 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.886914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:53755 (local address 127.2.208.129:36649)
0620 19:00:50.887032 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:50.887038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:50.887213 (+ 175us) server_negotiation.cc:407] Connection header received
0620 19:00:50.887390 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:50.887393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:50.887438 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:50.887522 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:00:50.887759 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.888428 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.889508 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:50.889743 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:50.889847 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:50.890098 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:00:50.890180 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:00:50.890263 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:50.890565 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 19:00:50.890613 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:00:50.909725 10413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.909868 10413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:00:50.911731 10413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c92a5cd859e4423ba9c4583af3e5174: Sending DeleteTablet for 3 replicas of tablet 56dfb0da172d47a8a4957fc8c9810261
I20260620 19:00:50.912276 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:48218
I20260620 19:00:50.912308 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58670
I20260620 19:00:50.912454 10994 tablet_replica.cc:330] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: stopping tablet replica
I20260620 19:00:50.912473 10997 tablet_replica.cc:330] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: stopping tablet replica
I20260620 19:00:50.912459 10831 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dfb0da172d47a8a4957fc8c9810261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:00:50 UTC) from {username='oryx', principal='oryx/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 19:00:50.912586 10994 raft_consensus.cc:2229] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:00:50.912612 10997 raft_consensus.cc:2229] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.912628 10995 tablet_replica.cc:330] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: stopping tablet replica
I20260620 19:00:50.912714 10997 raft_consensus.cc:2258] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.912715 10995 raft_consensus.cc:2229] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:00:50.912766 10994 raft_consensus.cc:2258] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.912802 10995 raft_consensus.cc:2258] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:00:50.913390 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 10486
I20260620 19:00:50.913610 10995 ts_tablet_manager.cc:1905] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.913607 10994 ts_tablet_manager.cc:1905] T 56dfb0da172d47a8a4957fc8c9810261 P 42db8fe454ea40bd9808a7c0591d5faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.915961 10995 ts_tablet_manager.cc:1918] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:50.916045 10995 log.cc:1196] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/56dfb0da172d47a8a4957fc8c9810261
I20260620 19:00:50.916258 10997 ts_tablet_manager.cc:1905] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:00:50.916298 10995 ts_tablet_manager.cc:1939] T 56dfb0da172d47a8a4957fc8c9810261 P 2f98425cbf32488886b70bf611392b7b: Deleting consensus metadata
I20260620 19:00:50.916899 10400 catalog_manager.cc:4841] TS 2f98425cbf32488886b70bf611392b7b (127.2.208.131:43783): tablet 56dfb0da172d47a8a4957fc8c9810261 (table test-table [id=16ceb22ccb704246b51e8d581b8aa3c3]) successfully deleted
I20260620 19:00:50.918295 10997 ts_tablet_manager.cc:1918] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:00:50.918386 10997 log.cc:1196] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/56dfb0da172d47a8a4957fc8c9810261
I20260620 19:00:50.918638 10997 ts_tablet_manager.cc:1939] T 56dfb0da172d47a8a4957fc8c9810261 P ce83830b1a8c4f788ca30023534bbd74: Deleting consensus metadata
I20260620 19:00:50.919234 10399 catalog_manager.cc:4841] TS ce83830b1a8c4f788ca30023534bbd74 (127.2.208.130:43191): tablet 56dfb0da172d47a8a4957fc8c9810261 (table test-table [id=16ceb22ccb704246b51e8d581b8aa3c3]) successfully deleted
W20260620 19:00:50.922693 10398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:36649 (error 108)
I20260620 19:00:50.922752 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 10626
I20260620 19:00:50.924352 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:50.923897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:36649 (local address 127.0.0.1:48238)
0620 19:00:50.924148 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:50.924228 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:36649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
W20260620 19:00:50.924602 10398 catalog_manager.cc:4584] TS 42db8fe454ea40bd9808a7c0591d5faa (127.2.208.129:36649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56dfb0da172d47a8a4957fc8c9810261: Network error: Client connection negotiation failed: client connection to 127.2.208.129:36649: connect: Connection refused (error 111)
I20260620 19:00:50.930119 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 10766
I20260620 19:00:50.936566 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 10382
2026-06-20T19:00:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3420 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[11039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[11039](info): set up 2 sockets
Jun 20 19:00:50 dist-test-slave-tsm4 krb5kdc[11039](info): commencing operation
krb5kdc: starting...
W20260620 19:00:51.968235 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:52 dist-test-slave-tsm4 krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:52.082130 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38563
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:52.188560 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:52.188817 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:52.188855 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:52.192768 11052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:52.192818 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:52.192837 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:52.192854 11052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:52.192910 11052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:52.197955 11052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:52.198920 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:52.199896 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:52.208086 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:52.208046 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:52.208209 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:52.208230 11052 server_base.cc:1034] running on GCE node
I20260620 19:00:52.208847 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:52.209858 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:52.211011 11052 hybrid_clock.cc:648] HybridClock initialized: now 1781982052210979 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:00:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1281 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:00:52 dist-test-slave-tsm4 krb5kdc[11065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:00:52 dist-test-slave-tsm4 krb5kdc[11065](info): set up 2 sockets
Jun 20 19:00:52 dist-test-slave-tsm4 krb5kdc[11065](info): commencing operation
krb5kdc: starting...
W20260620 19:00:53.244131 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:00:53 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:00:53.361754 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45669
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:45669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:53.469185 11078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:53.469440 11078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:53.469480 11078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:53.473412 11078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:53.473460 11078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:53.473479 11078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:53.473495 11078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:53.473553 11078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:53.478739 11078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:45669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:53.479732 11078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:53.480711 11078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:53.488704 11083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:53.488771 11086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:53.488704 11084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:53.488915 11078 server_base.cc:1034] running on GCE node
I20260620 19:00:53.489552 11078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:53.490604 11078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:53.491822 11078 hybrid_clock.cc:648] HybridClock initialized: now 1781982053491802 us; error 30 us; skew 500 ppm
Jun 20 19:00:53 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:53.499683 11078 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:53.500557 11078 webserver.cc:460] Webserver started at http://127.2.208.190:38905/ using document root <none> and password file <none>
I20260620 19:00:53.501127 11078 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:53.501217 11078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:53.501485 11078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:53.503577 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "1bedb8ce3feb4df2a750f096a327144f"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "f72afa6d2d1dc296597df07265a26a77"
server_key_iv: "2b6ddb697e1608be7c4e726404ad9762"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.504148 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "1bedb8ce3feb4df2a750f096a327144f"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "f72afa6d2d1dc296597df07265a26a77"
server_key_iv: "2b6ddb697e1608be7c4e726404ad9762"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.507812 11078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:53.510161 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:53.511303 11078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:53.511459 11078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "1bedb8ce3feb4df2a750f096a327144f"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "f72afa6d2d1dc296597df07265a26a77"
server_key_iv: "2b6ddb697e1608be7c4e726404ad9762"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.511567 11078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:53.556888 11078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:53.560117 11078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:53.560318 11078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:53.566967 11078 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:45669
I20260620 19:00:53.567054 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:45669 every 8 connection(s)
I20260620 19:00:53.567968 11078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:53.570626 11146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:00:53.575902 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Bootstrap starting.
I20260620 19:00:53.578167 11146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Neither blocks nor log segments found. Creating new log.
I20260620 19:00:53.578367 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11078
I20260620 19:00:53.578496 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:00:53.578763 2882 external_mini_cluster.cc:1428] Setting key dd00d0470737e8bc7357da584f88405d
I20260620 19:00:53.579007 11146 log.cc:824] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:53.580921 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: No bootstrap required, opened a new log
I20260620 19:00:53.584157 11146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:53.584400 11146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:00:53.584476 11146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bedb8ce3feb4df2a750f096a327144f, State: Initialized, Role: FOLLOWER
I20260620 19:00:53.584887 11146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:53.585026 11146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:53.585090 11146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:53.585215 11146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:00:53.586095 11146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:53.586417 11146 leader_election.cc:302] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bedb8ce3feb4df2a750f096a327144f; no voters:
I20260620 19:00:53.586700 11146 leader_election.cc:288] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:00:53.586826 11151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 19:00:53 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:53.587049 11151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 LEADER]: Becoming Leader. State: Replica: 1bedb8ce3feb4df2a750f096a327144f, State: Running, Role: LEADER
I20260620 19:00:53.587387 11151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:53.588037 11146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:53.588581 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bedb8ce3feb4df2a750f096a327144f. Latest consensus state: current_term: 1 leader_uuid: "1bedb8ce3feb4df2a750f096a327144f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } } }
I20260620 19:00:53.588676 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:53.589047 11156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:53.591563 11152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bedb8ce3feb4df2a750f096a327144f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } } }
I20260620 19:00:53.591681 11152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:53.593231 11156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:53.593606 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:53.579847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.2.208.190:45669)
0620 19:00:53.580270 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:53.580288 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:53.580330 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:00:53.580942 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:53.580957 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:53.581241 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:53.581562 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:53.582302 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.583225 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.584089 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.584686 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.584863 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:53.587607 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:53.587612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:53.587627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:53.587654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:53.590324 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:53.590818 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:53.590825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:53.590830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:53.590989 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:53.591286 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:53.591289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:53.591291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:53.591792 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:53.591991 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:53.592432 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 19:00:53.592712 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260620 19:00:53.599596 11156 catalog_manager.cc:1347] Generated new cluster ID: 5ac1e3da875c4898a2c32a62af09d96c
I20260620 19:00:53.599658 11156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:53.638417 11156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:00:53.639381 11156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:53.645179 11156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Generated new TSK 0
I20260620 19:00:53.645954 11156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:00:53.681756 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:53.789021 11172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:53.789256 11172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:53.789299 11172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:53.793272 11172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:53.793322 11172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:53.793377 11172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:53.798420 11172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:53.799340 11172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:53.800354 11172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:53.809793 11177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:53.809796 11178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:53.809939 11172 server_base.cc:1034] running on GCE node
W20260620 19:00:53.809859 11180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:53.810397 11172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:53.810998 11172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:53.812189 11172 hybrid_clock.cc:648] HybridClock initialized: now 1781982053812170 us; error 33 us; skew 500 ppm
Jun 20 19:00:53 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:53.814797 11172 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:00:53.815730 11172 webserver.cc:460] Webserver started at http://127.2.208.129:38645/ using document root <none> and password file <none>
I20260620 19:00:53.816238 11172 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:53.816293 11172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:53.816445 11172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:53.818374 11172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "6ffb43df9b4546609fc112e8d5f6b1b1"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "96a469b65ab9f8fd0055d023569cb24a"
server_key_iv: "da1e63beae48cad70ad5c3393256c2a8"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.818833 11172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "6ffb43df9b4546609fc112e8d5f6b1b1"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "96a469b65ab9f8fd0055d023569cb24a"
server_key_iv: "da1e63beae48cad70ad5c3393256c2a8"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.822289 11172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:00:53.824487 11187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:53.825593 11172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:00:53.825711 11172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "6ffb43df9b4546609fc112e8d5f6b1b1"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "96a469b65ab9f8fd0055d023569cb24a"
server_key_iv: "da1e63beae48cad70ad5c3393256c2a8"
server_key_version: "encryptionkey@0"
I20260620 19:00:53.825802 11172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:53.866726 11172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:53.869604 11172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:53.869803 11172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:53.870393 11172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:53.871202 11172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:53.871245 11172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:53.871330 11172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:53.871379 11172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:53.879861 11172 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:46525
I20260620 19:00:53.879890 11300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:46525 every 8 connection(s)
I20260620 19:00:53.880726 11172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:53.888108 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11172
I20260620 19:00:53.888245 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:00:53.888576 2882 external_mini_cluster.cc:1428] Setting key bc8e439c7093d2d72a7ffa097cb69860
Jun 20 19:00:53 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:53.894605 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:53.882413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37901 (local address 127.2.208.190:45669)
0620 19:00:53.882572 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:53.882577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:53.883482 (+ 905us) server_negotiation.cc:407] Connection header received
0620 19:00:53.884666 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:53.884668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:53.884713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:53.884785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:53.885980 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.886504 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.887634 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.887913 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.888018 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:53.890875 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:53.890881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:53.890887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:53.890918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:53.892421 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:53.893026 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:53.893028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:53.893029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:53.893097 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:53.893472 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:53.893474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:53.893475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:53.893627 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:53.893698 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:53.894378 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 19:00:53.894492 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:00:53.895871 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:53.882843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:37901)
0620 19:00:53.883276 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:53.883318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:53.884474 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:53.884949 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:53.884955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:53.885297 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:53.885856 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:53.885870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.886604 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.886606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:53.887535 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:53.887544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:53.888079 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:53.888083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:53.888282 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:53.889005 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:53.889021 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:53.890707 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:53.892533 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:53.892540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:53.892555 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:53.892929 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:53.893185 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:53.893190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:53.893195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:53.893378 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:53.893732 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:53.893741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:53.894076 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 19:00:53.894328 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 19:00:53.894680 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260620 19:00:53.897159 11301 heartbeater.cc:347] Connected to a master server at 127.2.208.190:45669
I20260620 19:00:53.897411 11301 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:53.897884 11301 heartbeater.cc:511] Master 127.2.208.190:45669 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:00:53.899405 11110 ts_manager.cc:194] Registered new tserver with Master: 6ffb43df9b4546609fc112e8d5f6b1b1 (127.2.208.129:46525)
I20260620 19:00:53.900686 11110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:37901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:00:53.915535 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:54.024519 11306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:54.024762 11306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:54.024803 11306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:54.028796 11306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:54.028851 11306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:54.028940 11306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:54.034162 11306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:54.035079 11306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:54.036108 11306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:54.045256 11311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.045274 11314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.045352 11312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.045452 11306 server_base.cc:1034] running on GCE node
I20260620 19:00:54.045997 11306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:54.046548 11306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:54.047708 11306 hybrid_clock.cc:648] HybridClock initialized: now 1781982054047696 us; error 24 us; skew 500 ppm
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:54.050421 11306 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:00:54.051275 11306 webserver.cc:460] Webserver started at http://127.2.208.130:39085/ using document root <none> and password file <none>
I20260620 19:00:54.051824 11306 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:54.051880 11306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:54.052029 11306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:54.053980 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "109d726405c8490c8715976cbb9c621c"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "438212858b3781d5f955a40224d4225b"
server_key_iv: "60d171a5c28506d69c0be2ca71d4fc95"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.054462 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "109d726405c8490c8715976cbb9c621c"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "438212858b3781d5f955a40224d4225b"
server_key_iv: "60d171a5c28506d69c0be2ca71d4fc95"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.057976 11306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:00:54.060315 11321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.061748 11306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:54.061873 11306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "109d726405c8490c8715976cbb9c621c"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "438212858b3781d5f955a40224d4225b"
server_key_iv: "60d171a5c28506d69c0be2ca71d4fc95"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.061981 11306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:54.085114 11306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:54.088430 11306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:54.088615 11306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:54.089293 11306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:54.090097 11306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:54.090176 11306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.090230 11306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:54.090268 11306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.099041 11306 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:36757
I20260620 19:00:54.099062 11434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:36757 every 8 connection(s)
I20260620 19:00:54.099969 11306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:54.101411 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11306
I20260620 19:00:54.101531 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:00:54.101823 2882 external_mini_cluster.cc:1428] Setting key 69a838afa11dabffd37f8e280efe0871
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
I20260620 19:00:54.114980 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.101692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:40321 (local address 127.2.208.190:45669)
0620 19:00:54.101821 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:54.101825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:54.102764 (+ 939us) server_negotiation.cc:407] Connection header received
0620 19:00:54.104165 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:54.104168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:54.104216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:54.104290 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:54.105882 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.106416 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.108236 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.108448 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.108528 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.111348 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:54.111356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:54.111392 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:54.111428 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:54.112926 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:54.113370 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:54.113373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:54.113374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:54.113450 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:54.113738 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:54.113742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:54.113743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:54.114029 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:54.114110 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:54.114652 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 19:00:54.114790 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:00:54.116039 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.102173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:40321)
0620 19:00:54.102635 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:54.102675 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:54.103955 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:54.104491 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:54.104499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:54.105015 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:54.105729 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.105754 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.106548 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.106555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.108118 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.108131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.108608 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.108616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.108860 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.109519 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:54.109547 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:54.111175 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:54.113024 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:54.113029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:54.113040 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:54.113284 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:54.113530 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:54.113532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:54.113533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:54.113651 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:54.114125 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:54.114130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:54.114389 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 19:00:54.114575 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:00:54.114895 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
I20260620 19:00:54.117218 11435 heartbeater.cc:347] Connected to a master server at 127.2.208.190:45669
I20260620 19:00:54.117460 11435 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:54.117930 11435 heartbeater.cc:511] Master 127.2.208.190:45669 requested a full tablet report, sending...
I20260620 19:00:54.119004 11110 ts_manager.cc:194] Registered new tserver with Master: 109d726405c8490c8715976cbb9c621c (127.2.208.130:36757)
I20260620 19:00:54.119590 11110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:40321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:00:54.130002 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:54.237388 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:54.237677 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:54.237749 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:54.241712 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:54.241793 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:54.241879 11440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:54.246960 11440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:54.248000 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:54.248984 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:00:54.258628 11440 server_base.cc:1034] running on GCE node
W20260620 19:00:54.258667 11445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.258594 11448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.258584 11446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.259280 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:54.259878 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:54.261055 11440 hybrid_clock.cc:648] HybridClock initialized: now 1781982054261041 us; error 29 us; skew 500 ppm
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:54.263808 11440 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:00:54.264545 11440 webserver.cc:460] Webserver started at http://127.2.208.131:45471/ using document root <none> and password file <none>
I20260620 19:00:54.265125 11440 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:54.265180 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:54.265398 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:00:54.267515 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "6605f5278e78422c9f5ae8c2944026dd"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "bd8d405f6e67caea762cacf44258abf0"
server_key_iv: "e1f88daae991cf80909aa6d6fbcd1b10"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.268079 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "6605f5278e78422c9f5ae8c2944026dd"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "bd8d405f6e67caea762cacf44258abf0"
server_key_iv: "e1f88daae991cf80909aa6d6fbcd1b10"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.271556 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:00:54.273711 11455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.274824 11440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:54.274912 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "6605f5278e78422c9f5ae8c2944026dd"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "bd8d405f6e67caea762cacf44258abf0"
server_key_iv: "e1f88daae991cf80909aa6d6fbcd1b10"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.275055 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:54.316066 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:54.319386 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:54.319605 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:54.320220 11440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:54.321003 11440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:54.321053 11440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.321138 11440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:54.321171 11440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.329764 11440 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:46023
I20260620 19:00:54.329775 11568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:46023 every 8 connection(s)
I20260620 19:00:54.330773 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:54.337028 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11440
I20260620 19:00:54.337170 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:00:54.337479 2882 external_mini_cluster.cc:1428] Setting key 97a76a75444de0c05c0686de687281da
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:00:54.343966 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.332663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:42919 (local address 127.2.208.190:45669)
0620 19:00:54.332797 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:54.332801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:54.333789 (+ 988us) server_negotiation.cc:407] Connection header received
0620 19:00:54.334790 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:54.334793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:54.334839 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:54.334911 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:54.336104 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.336681 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.338225 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.338546 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.338661 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.341129 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:54.341131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:54.341134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:54.341155 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:54.342328 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:54.342724 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:54.342726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:54.342727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:54.342790 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:54.343045 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:54.343047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:54.343048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:54.343217 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:54.343293 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:54.343769 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 19:00:54.343857 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:00:54.344715 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.333020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:42919)
0620 19:00:54.333632 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:54.333672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:54.334609 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:54.335074 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:54.335080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:54.335433 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:54.335914 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.335927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.336818 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.336824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.338114 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.338123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.338728 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.338733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.338931 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.339608 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:54.339628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:54.340993 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:00:54.342418 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:54.342422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:54.342433 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:54.342648 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:54.342858 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:00:54.342860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:00:54.342861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:00:54.342978 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:00:54.343301 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:00:54.343305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:00:54.343570 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 19:00:54.343726 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 19:00:54.343947 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":104,"threads_started":1}
I20260620 19:00:54.345685 11569 heartbeater.cc:347] Connected to a master server at 127.2.208.190:45669
I20260620 19:00:54.345901 11569 heartbeater.cc:464] Registering TS with master...
I20260620 19:00:54.346359 11569 heartbeater.cc:511] Master 127.2.208.190:45669 requested a full tablet report, sending...
I20260620 19:00:54.347270 11110 ts_manager.cc:194] Registered new tserver with Master: 6605f5278e78422c9f5ae8c2944026dd (127.2.208.131:46023)
I20260620 19:00:54.347824 11110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:42919
I20260620 19:00:54.351039 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
Entry for principal oryx/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab.
WARNING: no policy specified for oryx/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
Entry for principal oryx/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab.
WARNING: no policy specified for oryx/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
Entry for principal oryx/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab.
I20260620 19:00:54.417641 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11172
I20260620 19:00:54.423098 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11306
I20260620 19:00:54.428220 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11440
I20260620 19:00:54.433305 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11078
I20260620 19:00:54.439706 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45669
--webserver_interface=127.2.208.190
--webserver_port=38905
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:45669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:00:54.545351 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:54.545634 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:54.545709 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:54.549753 11578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:00:54.549831 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:54.549870 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:54.549925 11578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:00:54.549961 11578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:00:54.554967 11578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:45669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:45669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:54.555984 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:54.556960 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:54.564989 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.564989 11584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.565088 11578 server_base.cc:1034] running on GCE node
W20260620 19:00:54.565035 11586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.565781 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:54.566890 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:54.568065 11578 hybrid_clock.cc:648] HybridClock initialized: now 1781982054568056 us; error 33 us; skew 500 ppm
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:54.570721 11578 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:00:54.571601 11578 webserver.cc:460] Webserver started at http://127.2.208.190:38905/ using document root <none> and password file <none>
I20260620 19:00:54.572124 11578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:54.572228 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:54.575780 11578 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 19:00:54.577764 11593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.578897 11578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:54.579038 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "1bedb8ce3feb4df2a750f096a327144f"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "f72afa6d2d1dc296597df07265a26a77"
server_key_iv: "2b6ddb697e1608be7c4e726404ad9762"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.579603 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:54.588246 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:54.591284 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:54.591547 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:54.598881 11578 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:45669
I20260620 19:00:54.598953 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:45669 every 8 connection(s)
I20260620 19:00:54.599869 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:00:54.603899 11646 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:00:54.605165 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11578
I20260620 19:00:54.605578 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:46525
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:45669
--webserver_port=38645
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:00:54.605602 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Bootstrap starting.
I20260620 19:00:54.622179 11646 log.cc:824] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:00:54.628491 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:00:54.628847 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Bootstrap complete.
I20260620 19:00:54.633081 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:54.633512 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bedb8ce3feb4df2a750f096a327144f, State: Initialized, Role: FOLLOWER
I20260620 19:00:54.634023 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:54.634171 11646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:00:54.634269 11646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:00:54.634377 11646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:00:54.636308 11646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:54.636626 11646 leader_election.cc:302] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bedb8ce3feb4df2a750f096a327144f; no voters:
I20260620 19:00:54.636948 11646 leader_election.cc:288] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:00:54.637028 11650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:00:54.637249 11650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [term 2 LEADER]: Becoming Leader. State: Replica: 1bedb8ce3feb4df2a750f096a327144f, State: Running, Role: LEADER
I20260620 19:00:54.637553 11650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } }
I20260620 19:00:54.638240 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:00:54.638477 11652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1bedb8ce3feb4df2a750f096a327144f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } } }
I20260620 19:00:54.638597 11652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:54.638840 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bedb8ce3feb4df2a750f096a327144f. Latest consensus state: current_term: 2 leader_uuid: "1bedb8ce3feb4df2a750f096a327144f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bedb8ce3feb4df2a750f096a327144f" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 45669 } } }
I20260620 19:00:54.638931 11654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:00:54.638937 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f [sys.catalog]: This master's current role is: LEADER
I20260620 19:00:54.642211 11654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:00:54.642820 11654 catalog_manager.cc:1259] Loaded cluster ID: 5ac1e3da875c4898a2c32a62af09d96c
I20260620 19:00:54.642877 11654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:00:54.645167 11654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:00:54.645802 11654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1bedb8ce3feb4df2a750f096a327144f: Loaded TSK: 0
I20260620 19:00:54.646958 11654 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:00:54.719298 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:54.719640 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:54.719720 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:54.723639 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:54.723712 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:54.723805 11648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:00:54.729009 11648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.129.keytab
--principal=oryx/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:46525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=38645
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:54.729992 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:54.730978 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:54.741122 11673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.741222 11648 server_base.cc:1034] running on GCE node
W20260620 19:00:54.741133 11675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.741423 11672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.741997 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:54.742749 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:54.743966 11648 hybrid_clock.cc:648] HybridClock initialized: now 1781982054743946 us; error 35 us; skew 500 ppm
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:54.747547 11648 init.cc:377] Logged in from keytab as oryx/127.2.208.129@KRBTEST.COM (short username oryx)
I20260620 19:00:54.748651 11648 webserver.cc:460] Webserver started at http://127.2.208.129:38645/ using document root <none> and password file <none>
I20260620 19:00:54.749321 11648 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:54.749413 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:54.754261 11648 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:00:54.756711 11682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.758157 11648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:00:54.758267 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "6ffb43df9b4546609fc112e8d5f6b1b1"
format_stamp: "Formatted at 2026-06-20 19:00:53 on dist-test-slave-tsm4"
server_key: "96a469b65ab9f8fd0055d023569cb24a"
server_key_iv: "da1e63beae48cad70ad5c3393256c2a8"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.758936 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:54.776501 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:54.779860 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:54.780077 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:54.780792 11648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:54.781817 11648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:54.781875 11648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.781924 11648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:54.781955 11648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.792896 11648 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:46525
I20260620 19:00:54.792948 11795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:46525 every 8 connection(s)
I20260620 19:00:54.793872 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:00:54.802399 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11648
I20260620 19:00:54.802977 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:36757
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:45669
--webserver_port=39085
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:00:54.809329 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:54.795955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55445 (local address 127.2.208.190:45669)
0620 19:00:54.796374 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:54.796394 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:54.796989 (+ 595us) server_negotiation.cc:407] Connection header received
0620 19:00:54.798081 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:54.798097 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:54.798367 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:54.798674 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:54.800005 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.801838 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.803632 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.804389 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.804565 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.808561 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:55445: BlockingRecv error: recv got EOF from 127.2.208.129:55445 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260620 19:00:54.809338 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.796188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:55445)
0620 19:00:54.796829 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:54.796868 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:54.797695 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:54.798745 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:54.798753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:54.799155 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:54.799778 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.799814 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.802004 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.802008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.803477 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.803491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.804829 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.804835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.805036 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.805735 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:54.805757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:54.808365 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":127,"threads_started":1}
W20260620 19:00:54.809774 11796 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:45669 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.208.190:45669: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:54.815073 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.810252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:55653)
0620 19:00:54.810361 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:54.810375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:54.810454 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:54.810698 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:54.810701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:54.810860 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:54.810940 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.810945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.811729 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.811735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.812271 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.812279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.812719 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.812723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.812814 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.813305 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:54.813316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:54.814933 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:00:54.815176 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:54.810299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55653 (local address 127.2.208.190:45669)
0620 19:00:54.810401 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:54.810407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:54.810421 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:00:54.810547 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:54.810550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:54.810592 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:54.810685 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:54.811046 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.811630 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.812386 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.812614 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.812719 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.815045 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:55653: BlockingRecv error: recv got EOF from 127.2.208.129:55653 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:54.833302 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:54.819293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:53021)
0620 19:00:54.819467 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:54.819478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:54.819534 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:54.819813 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:54.819815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:54.819966 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:54.820027 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.820031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.820717 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.820720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.821220 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:54.821226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.821643 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.821647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:54.821723 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.822200 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:54.822212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:54.833151 (+ 10939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260620 19:00:54.833395 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:54.819344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:53021 (local address 127.2.208.190:45669)
0620 19:00:54.819642 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:54.819645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:54.819654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:00:54.819679 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:54.819681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:54.819713 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:54.819799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:54.820136 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.820607 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.821329 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:54.821534 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:54.821646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:54.833276 (+ 11630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:53021: BlockingRecv error: recv got EOF from 127.2.208.129:53021 (error 108)
Metrics: {"server-negotiator.queue_time_us":239}
W20260620 19:00:54.833763 11796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:00:54.940713 11800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:54.940961 11800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:54.941032 11800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:54.944952 11800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:54.945010 11800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:54.945068 11800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:00:54.950388 11800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.130.keytab
--principal=oryx/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:36757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=39085
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:54.951485 11800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:54.952461 11800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:54.961556 11806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.961766 11800 server_base.cc:1034] running on GCE node
W20260620 19:00:54.961820 11805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:54.961808 11808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:54.962292 11800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:54.962920 11800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:54.964087 11800 hybrid_clock.cc:648] HybridClock initialized: now 1781982054964080 us; error 30 us; skew 500 ppm
Jun 20 19:00:54 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982054, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:54.966701 11800 init.cc:377] Logged in from keytab as oryx/127.2.208.130@KRBTEST.COM (short username oryx)
I20260620 19:00:54.967592 11800 webserver.cc:460] Webserver started at http://127.2.208.130:39085/ using document root <none> and password file <none>
I20260620 19:00:54.968144 11800 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:54.968235 11800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:54.971935 11800 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 19:00:54.973882 11815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.975036 11800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:00:54.975148 11800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "109d726405c8490c8715976cbb9c621c"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "438212858b3781d5f955a40224d4225b"
server_key_iv: "60d171a5c28506d69c0be2ca71d4fc95"
server_key_version: "encryptionkey@0"
I20260620 19:00:54.975708 11800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:54.993263 11800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:54.995990 11800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:54.996183 11800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:54.996768 11800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:54.997541 11800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:54.997584 11800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:54.997650 11800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:54.997687 11800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:55.006342 11800 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:36757
I20260620 19:00:55.006414 11928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:36757 every 8 connection(s)
I20260620 19:00:55.007311 11800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:00:55.013703 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11800
I20260620 19:00:55.014127 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:46023
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:45669
--webserver_port=45471
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:00:55.018421 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.009070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:55191 (local address 127.2.208.190:45669)
0620 19:00:55.009198 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.009202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.009989 (+ 787us) server_negotiation.cc:407] Connection header received
0620 19:00:55.011238 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.011241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.011299 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.011450 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.012549 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.013071 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.014296 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.014721 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.014842 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.018299 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:55191: BlockingRecv error: recv got EOF from 127.2.208.130:55191 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:00:55.019021 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.009451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:55191)
0620 19:00:55.009855 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.009894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.011031 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.011615 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.011622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.011955 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.012407 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.012419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.013192 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.013196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.014140 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.014149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.014879 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.014882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.015015 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.015679 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.015694 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.018189 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
W20260620 19:00:55.019584 11929 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:45669 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.208.190:45669: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:55.025409 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.020040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:58329)
0620 19:00:55.020250 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.020261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.020321 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.020560 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.020562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.020676 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.020755 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.020759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.021788 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.021790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.022294 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.022300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.022863 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.022866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.022959 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.023524 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.023535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.025323 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260620 19:00:55.025597 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.020145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:58329 (local address 127.2.208.190:45669)
0620 19:00:55.020300 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.020305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.020322 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:00:55.020390 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.020396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.020451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.020542 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.020859 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.021687 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.022394 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.022763 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.022878 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.025479 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:58329: BlockingRecv error: recv got EOF from 127.2.208.130:58329 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:55.030956 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.026073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:43859)
0620 19:00:55.026185 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.026202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.026258 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.026507 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.026509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.026630 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.026709 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.026714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.027699 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.027701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.028216 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.028224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.028732 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.028734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.028784 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.029257 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.029267 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.030867 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:00:55.031111 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.026139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:43859 (local address 127.2.208.190:45669)
0620 19:00:55.026236 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.026240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.026253 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:55.026337 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.026339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.026377 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.026457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.026813 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.027604 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.028309 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.028634 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.028730 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.031003 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:43859: BlockingRecv error: recv got EOF from 127.2.208.130:43859 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 19:00:55.031147 11929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:00:55.127945 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:00:55.128176 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:00:55.128216 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:00:55.132169 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:00:55.132220 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:00:55.132273 11932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:00:55.137233 11932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/krb5kdc/oryx_127.2.208.131.keytab
--principal=oryx/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:46023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=45471
--tserver_master_addrs=127.2.208.190:45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:00:55.138294 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:00:55.139534 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:00:55.149024 11938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:55.149096 11940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:00:55.149008 11937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:00:55.149186 11932 server_base.cc:1034] running on GCE node
I20260620 19:00:55.149659 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:00:55.150354 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:00:55.151587 11932 hybrid_clock.cc:648] HybridClock initialized: now 1781982055151556 us; error 44 us; skew 500 ppm
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982055, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:00:55.154230 11932 init.cc:377] Logged in from keytab as oryx/127.2.208.131@KRBTEST.COM (short username oryx)
I20260620 19:00:55.155025 11932 webserver.cc:460] Webserver started at http://127.2.208.131:45471/ using document root <none> and password file <none>
I20260620 19:00:55.155580 11932 fs_manager.cc:359] Metadata directory not provided
I20260620 19:00:55.155649 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:00:55.159139 11932 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:00:55.161069 11947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:55.162230 11932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 19:00:55.162362 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "6605f5278e78422c9f5ae8c2944026dd"
format_stamp: "Formatted at 2026-06-20 19:00:54 on dist-test-slave-tsm4"
server_key: "bd8d405f6e67caea762cacf44258abf0"
server_key_iv: "e1f88daae991cf80909aa6d6fbcd1b10"
server_key_version: "encryptionkey@0"
I20260620 19:00:55.162873 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:00:55.177382 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:00:55.180751 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:00:55.180939 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:00:55.181516 11932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:00:55.182340 11932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:00:55.182403 11932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:55.182466 11932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:00:55.182518 11932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:00:55.191248 11932 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:46023
I20260620 19:00:55.191289 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:46023 every 8 connection(s)
I20260620 19:00:55.192301 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:00:55.200927 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 11932
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:00:55.203610 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.194071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:43833 (local address 127.2.208.190:45669)
0620 19:00:55.194197 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.194202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.195023 (+ 821us) server_negotiation.cc:407] Connection header received
0620 19:00:55.196350 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.196353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.196420 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.196503 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.197629 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.198177 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.199294 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.199600 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.199727 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.203489 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:43833: BlockingRecv error: recv got EOF from 127.2.208.131:43833 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:00:55.204286 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.194412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:43833)
0620 19:00:55.194848 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.194890 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.196141 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.196680 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.196686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.197025 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.197487 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.197499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.198297 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.198300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.199161 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.199169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.199762 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.199766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.199933 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.200534 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.200549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.203336 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
W20260620 19:00:55.204694 12061 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:45669 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.208.190:45669: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:55.210244 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.205125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:54831)
0620 19:00:55.205217 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.205229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.205286 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.205514 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.205518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.205675 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.205765 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.205770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.206468 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.206471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.207152 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.207163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.207759 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.207766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.207868 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.208185 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.208203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.210116 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:00:55.210426 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.205170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:54831 (local address 127.2.208.190:45669)
0620 19:00:55.205269 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.205272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.205283 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:00:55.205381 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.205383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.205413 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.205473 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.205867 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.206338 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.207304 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.207649 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.207725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.210215 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:54831: BlockingRecv error: recv got EOF from 127.2.208.131:54831 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:00:55.211362 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.201913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.2.208.190:45669)
0620 19:00:55.202254 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.202257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.202266 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:00:55.202293 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.202295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.202325 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.202475 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.203457 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.204018 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.205134 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.205474 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.205575 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.206914 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:00:55.206921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:00:55.206938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:00:55.206966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:00:55.208953 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:55.209457 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:55.209466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:55.209472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:55.209623 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:00:55.209963 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:00:55.209968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:00:55.209970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:00:55.210309 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:00:55.210515 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 19:00:55.210823 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 19:00:55.211188 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":70,"threads_started":1}
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:55.216040 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.210937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:43127)
0620 19:00:55.211082 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.211095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.211161 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.211375 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.211381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.211538 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.211611 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.211616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.212478 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.212481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.213216 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.213225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.213791 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.213797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.213854 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.214291 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.214301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.215919 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260620 19:00:55.216114 11798 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.211026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:43127 (local address 127.2.208.190:45669)
0620 19:00:55.211163 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.211166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.211178 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:55.211233 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.211235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.211264 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.211322 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.211709 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.212364 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.213355 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.213561 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.213670 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.216018 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:43127: BlockingRecv error: recv got EOF from 127.2.208.131:43127 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 19:00:55.216285 12061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:55 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:55.840294 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:55.834600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:44541)
0620 19:00:55.834889 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:55.834921 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:55.835031 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:55.835335 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:55.835341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:55.835610 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:55.835750 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.835760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.836504 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.836506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.837070 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:55.837077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.837479 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.837482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:55.837559 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.838138 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:55.838150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:55.840098 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260620 19:00:55.840366 12067 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:55.834666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:44541 (local address 127.2.208.190:45669)
0620 19:00:55.834980 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:55.834983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:55.835017 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:00:55.835138 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:55.835141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:55.835197 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:55.835282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:55.835875 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.836415 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.837166 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:55.837388 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:55.837463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:55.840221 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:44541: BlockingRecv error: recv got EOF from 127.2.208.129:44541 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:56.037410 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:56.031700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:39919)
0620 19:00:56.032033 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:56.032055 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:56.032206 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:56.032501 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:56.032504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:56.032738 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:56.032829 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.032834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.033607 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.033613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.034195 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.034202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.034663 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.034667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.034752 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.035317 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:56.035329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:56.037229 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
W20260620 19:00:56.037475 12067 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:56.031837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:39919 (local address 127.2.208.190:45669)
0620 19:00:56.031995 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:56.032000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:56.032141 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:00:56.032322 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:56.032326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:56.032390 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:56.032479 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:56.032960 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.033495 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.034318 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.034551 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.034661 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.037351 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:39919: BlockingRecv error: recv got EOF from 127.2.208.130:39919 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:56.222447 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:56.216866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:56879)
0620 19:00:56.217159 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:56.217171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:56.217300 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:56.217560 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:56.217563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:56.217764 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:56.217874 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.217880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.218621 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.218627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.219200 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.219207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.219739 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.219743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.219839 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.220360 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:56.220373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:56.222279 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260620 19:00:56.222512 12067 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:56.216954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:56879 (local address 127.2.208.190:45669)
0620 19:00:56.217108 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:56.217113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:56.217243 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:00:56.217396 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:56.217400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:56.217455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:56.217538 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:56.217990 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.218509 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.219321 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.219628 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.219736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.222404 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:56879: BlockingRecv error: recv got EOF from 127.2.208.131:56879 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:56 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:56.846441 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:56.841171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:55259)
0620 19:00:56.841514 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:56.841546 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:56.841674 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:56.841950 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:56.841952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:56.842122 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:56.842233 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.842238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.843011 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.843013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.843601 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:56.843608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.844024 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.844027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:56.844085 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.844543 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:56.844554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:56.846315 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":83,"threads_started":1}
W20260620 19:00:56.846648 12071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:56.841245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:55259 (local address 127.2.208.190:45669)
0620 19:00:56.841552 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:56.841555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:56.841634 (+ 79us) server_negotiation.cc:407] Connection header received
0620 19:00:56.841776 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:56.841778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:56.841824 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:56.841918 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:56.842344 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.842922 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.843702 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:56.843931 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:56.844012 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:56.846496 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:55259: BlockingRecv error: recv got EOF from 127.2.208.129:55259 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:57.044198 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:57.038482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:49549)
0620 19:00:57.038783 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:57.038802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:57.038929 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:57.039238 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:57.039242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:57.039488 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:57.039597 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.039604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.040404 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.040410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.040988 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.040996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.041488 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.041491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.041578 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.042143 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:57.042154 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:57.044001 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
W20260620 19:00:57.044234 12071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:57.038519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:49549 (local address 127.2.208.190:45669)
0620 19:00:57.038734 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:57.038740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:57.038903 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:00:57.039031 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:57.039034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:57.039089 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:57.039197 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:57.039715 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.040292 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.041106 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.041375 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.041487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.044132 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:49549: BlockingRecv error: recv got EOF from 127.2.208.130:49549 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:57.228782 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:57.223311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:53841)
0620 19:00:57.223678 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:57.223689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:57.223809 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:57.224096 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:57.224099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:57.224304 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:57.224389 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.224395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.225129 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.225134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.225695 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.225702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.226146 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.226150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.226231 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.226741 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:57.226752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:57.228670 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
W20260620 19:00:57.228963 12071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:57.223458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:53841 (local address 127.2.208.190:45669)
0620 19:00:57.223590 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:57.223595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:57.223757 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:00:57.223910 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:57.223914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:57.223985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:57.224080 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:57.224498 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.225012 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.225810 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.226029 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.226143 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.228816 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:53841: BlockingRecv error: recv got EOF from 127.2.208.131:53841 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:57 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:57.852603 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:57.847228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:37183)
0620 19:00:57.847541 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:57.847570 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:57.847673 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:57.847914 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:57.847916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:57.848086 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:57.848177 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.848183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.848957 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.848960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.849514 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:57.849521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.849923 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.849926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:57.849983 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.850454 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:57.850464 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:57.852474 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":137,"threads_started":1}
W20260620 19:00:57.852720 12075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:57.847299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37183 (local address 127.2.208.190:45669)
0620 19:00:57.847601 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:57.847605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:57.847618 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:00:57.847760 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:57.847762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:57.847804 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:57.847886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:57.848303 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.848827 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.849619 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:57.849832 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:57.849908 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:57.852595 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:37183: BlockingRecv error: recv got EOF from 127.2.208.129:37183 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:58.051208 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:58.045085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:52667)
0620 19:00:58.045449 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:58.045470 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:58.045604 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:58.045870 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:58.045874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:58.046085 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:58.046172 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.046177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.047213 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.047216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.048028 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.048037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.048484 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.048486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.048551 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.049118 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:58.049130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:58.051039 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
W20260620 19:00:58.051276 12075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:58.045235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:52667 (local address 127.2.208.190:45669)
0620 19:00:58.045410 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:58.045415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:58.045542 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:00:58.045709 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:58.045712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:58.045767 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:58.045850 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:58.046282 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.047088 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.048160 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.048370 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.048455 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.051168 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:52667: BlockingRecv error: recv got EOF from 127.2.208.130:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:58.235584 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:58.229688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:43589)
0620 19:00:58.230007 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:58.230020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:58.230150 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:58.230432 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:58.230435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:58.230647 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:58.230735 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.230741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.231558 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.231561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.232310 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.232320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.232841 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.232845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.232964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.233521 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:58.233542 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:58.235354 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
W20260620 19:00:58.235599 12075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:58.229769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:43589 (local address 127.2.208.190:45669)
0620 19:00:58.229947 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:58.229951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:58.230100 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:00:58.230259 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:58.230263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:58.230317 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:58.230406 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:58.230854 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.231441 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.232443 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.232726 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.232804 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.235499 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:43589: BlockingRecv error: recv got EOF from 127.2.208.131:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:58 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:58.864233 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:58.853572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:56583)
0620 19:00:58.853887 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:58.853910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:58.853999 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:58.854285 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:58.854288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:58.854489 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:58.854607 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.854616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.855415 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.855420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.855985 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:58.855992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.856476 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.856480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:58.856567 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.862056 (+ 5489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:58.862069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:58.864112 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
W20260620 19:00:58.864444 12079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:58.853636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:56583 (local address 127.2.208.190:45669)
0620 19:00:58.853915 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:58.853920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:58.853958 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:00:58.854095 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:58.854098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:58.854154 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:58.854261 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:58.854736 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.855273 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.856104 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:58.856361 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:58.856473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:58.864259 (+ 7786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:56583: BlockingRecv error: recv got EOF from 127.2.208.129:56583 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:59.057140 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:59.051972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:44901)
0620 19:00:59.052251 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:59.052263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:59.052360 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:59.052581 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:59.052583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:59.052759 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:59.052850 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.052855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.053597 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.053599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.054147 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.054154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.054551 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.054554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.054610 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.055164 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:59.055175 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:59.057026 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
W20260620 19:00:59.057284 12079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:59.052065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:44901 (local address 127.2.208.190:45669)
0620 19:00:59.052205 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:59.052208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:59.052330 (+ 122us) server_negotiation.cc:407] Connection header received
0620 19:00:59.052437 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:59.052439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:59.052481 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:59.052550 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:59.052964 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.053499 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.054241 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.054438 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.054528 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.057164 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:44901: BlockingRecv error: recv got EOF from 127.2.208.130:44901 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:59.241626 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:59.236392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:39759)
0620 19:00:59.236676 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:59.236689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:59.236785 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:59.236992 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:59.236994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:59.237164 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:59.237257 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.237262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.238024 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.238027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.238573 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.238580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.239002 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.239005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.239068 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.239674 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:59.239685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:59.241506 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
W20260620 19:00:59.241812 12079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:59.236416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39759 (local address 127.2.208.190:45669)
0620 19:00:59.236545 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:59.236549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:59.236750 (+ 201us) server_negotiation.cc:407] Connection header received
0620 19:00:59.236849 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:59.236851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:59.236894 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:59.236960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:59.237348 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.237931 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.238668 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.238910 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.238996 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.241644 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:39759: BlockingRecv error: recv got EOF from 127.2.208.131:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:00:59 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:00:59.870613 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:00:59.864957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:53429)
0620 19:00:59.865204 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 19:00:59.865227 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 19:00:59.865313 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:00:59.865580 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:00:59.865583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:00:59.865778 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:00:59.865899 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.865904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.866694 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.866697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.867289 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:00:59.867297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.867876 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.867879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:00:59.867943 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.868487 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:00:59.868499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:00:59.870445 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
W20260620 19:00:59.870668 12083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:00:59.865056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:53429 (local address 127.2.208.190:45669)
0620 19:00:59.865299 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 19:00:59.865302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:00:59.865314 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:00:59.865391 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:00:59.865393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:00:59.865443 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:00:59.865517 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:00:59.866025 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.866587 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.867512 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:00:59.867771 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:00:59.867848 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:00:59.870556 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:53429: BlockingRecv error: recv got EOF from 127.2.208.129:53429 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:00.063705 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:00.058046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:39501)
0620 19:01:00.058360 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:00.058377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:00.058512 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:00.058779 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:00.058782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:00.059014 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:00.059108 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.059114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.059986 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.059992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.060564 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.060572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.061060 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.061063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.061149 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.061708 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:00.061720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:00.063592 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
W20260620 19:01:00.063879 12083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:00.058213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:39501 (local address 127.2.208.190:45669)
0620 19:01:00.058377 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:00.058381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:00.058455 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:01:00.058610 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:00.058614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:00.058661 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:00.058753 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:00.059229 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.059879 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.060693 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.060950 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.061057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.063722 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:39501: BlockingRecv error: recv got EOF from 127.2.208.130:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:00.247856 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:00.242433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:40363)
0620 19:01:00.242700 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:00.242713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:00.242832 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:00.243071 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:00.243073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:00.243251 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:00.243344 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.243349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.244115 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.244118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.244694 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.244702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.245100 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.245102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.245163 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.245707 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:00.245719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:00.247735 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
W20260620 19:01:00.248013 12083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:00.242522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:40363 (local address 127.2.208.190:45669)
0620 19:01:00.242654 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:00.242657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:00.242780 (+ 123us) server_negotiation.cc:407] Connection header received
0620 19:01:00.242917 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:00.242919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:00.242963 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:00.243038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:00.243507 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.244010 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.244795 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.245001 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.245073 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.247856 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:40363: BlockingRecv error: recv got EOF from 127.2.208.131:40363 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:00 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:00.876776 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:00.871285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:59443)
0620 19:01:00.871588 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:00.871608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:00.871708 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:00.871967 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:00.871972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:00.872171 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:00.872264 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.872270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.873049 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.873052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.873623 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:00.873630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.874086 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.874089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:00.874167 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.874756 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:00.874768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:00.876659 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
W20260620 19:01:00.876965 12087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:00.871460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59443 (local address 127.2.208.190:45669)
0620 19:01:00.871701 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:00.871705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:00.871718 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:01:00.871772 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:00.871774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:00.871813 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:00.871904 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:00.872374 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.872951 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.873726 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:00.873963 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:00.874039 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:00.876803 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:59443: BlockingRecv error: recv got EOF from 127.2.208.129:59443 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:01.070122 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:01.064418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:57463)
0620 19:01:01.064695 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:01.064708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:01.064836 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:01.065136 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:01.065139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:01.065392 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:01.065536 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.065544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.066413 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.066419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.067037 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.067049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.067571 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.067575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.067659 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.068215 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:01.068229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:01.069948 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260620 19:01:01.070185 12087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:01.064523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:57463 (local address 127.2.208.190:45669)
0620 19:01:01.064692 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:01.064697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:01.064759 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:01:01.064957 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:01.064960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:01.065016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:01.065110 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:01.065627 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.066302 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.067137 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.067455 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.067571 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.070075 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:57463: BlockingRecv error: recv got EOF from 127.2.208.130:57463 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:01.254415 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:01.248602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:39463)
0620 19:01:01.248895 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:01.248915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:01.249040 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:01.249307 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:01.249310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:01.249517 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:01.249603 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.249609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.250664 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.250668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.251384 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.251394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.251810 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.251813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.251892 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.252354 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:01.252365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:01.254294 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
W20260620 19:01:01.254560 12087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:01.248665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39463 (local address 127.2.208.190:45669)
0620 19:01:01.248831 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:01.248836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:01.248986 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:01:01.249149 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:01.249154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:01.249203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:01.249288 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:01.249715 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.250529 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.251493 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.251715 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.251794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.254415 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:39463: BlockingRecv error: recv got EOF from 127.2.208.131:39463 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:01 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:01.883212 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:01.877603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:35787)
0620 19:01:01.877921 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:01.877934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:01.878028 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:01.878343 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:01.878346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:01.878608 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:01.878714 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.878721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.879506 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.879509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.880151 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:01.880160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.880585 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.880588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:01.880647 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.881125 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:01.881136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:01.883093 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
W20260620 19:01:01.883464 12091 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:01.877701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:35787 (local address 127.2.208.190:45669)
0620 19:01:01.877928 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:01.877932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:01.878002 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:01:01.878127 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:01.878129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:01.878189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:01.878273 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:01.878832 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.879384 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.880254 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:01.880480 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:01.880570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:01.883297 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:35787: BlockingRecv error: recv got EOF from 127.2.208.129:35787 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:02.076577 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:02.070919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:54749)
0620 19:01:02.071247 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:02.071261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:02.071397 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:02.071626 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:02.071628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:02.071803 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:02.071900 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.071905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.072748 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.072752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.073489 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.073501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.074029 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.074033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.074116 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.074605 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:02.074615 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:02.076461 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
W20260620 19:01:02.076959 12091 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:02.071001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:54749 (local address 127.2.208.190:45669)
0620 19:01:02.071136 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:02.071139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:02.071339 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:01:02.071477 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:02.071479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:02.071520 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:02.071592 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:02.072000 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.072627 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.073604 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.073909 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.074021 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.076669 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:54749: BlockingRecv error: recv got EOF from 127.2.208.130:54749 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:02.260496 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:02.255171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:44741)
0620 19:01:02.255534 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:02.255547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:02.255643 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:02.255886 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:02.255888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:02.256063 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:02.256154 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.256160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.256878 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.256881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.257466 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.257475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.257865 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.257867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.257927 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.258502 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:02.258517 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:02.260386 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
W20260620 19:01:02.260618 12091 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:02.255398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:44741 (local address 127.2.208.190:45669)
0620 19:01:02.255526 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:02.255529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:02.255610 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:01:02.255713 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:02.255715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:02.255759 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:02.255827 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:02.256246 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.256736 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.257567 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.257772 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.257843 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.260503 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:44741: BlockingRecv error: recv got EOF from 127.2.208.131:44741 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:02 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:02.889400 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:02.884091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:40481)
0620 19:01:02.884391 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:02.884409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:02.884546 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:02.884857 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:02.884860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:02.885118 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:02.885251 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.885259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.885996 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.885998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.886544 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:02.886551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.886943 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.886946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:02.887003 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.887573 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:02.887584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:02.889283 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260620 19:01:02.889537 12095 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:02.884234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:40481 (local address 127.2.208.190:45669)
0620 19:01:02.884476 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:02.884481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:02.884499 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:01:02.884656 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:02.884659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:02.884712 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:02.884809 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:02.885375 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.885906 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.886639 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:02.886848 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:02.886925 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:02.889407 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:40481: BlockingRecv error: recv got EOF from 127.2.208.129:40481 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:03.083777 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:03.077539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:48769)
0620 19:01:03.077826 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:03.077843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:03.077997 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:03.078276 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:03.078279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:03.078512 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:03.078624 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.078631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.079611 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.079617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.080227 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.080235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.080758 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.080761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.080838 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.081438 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:03.081452 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:03.083639 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
W20260620 19:01:03.083992 12095 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:03.077596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:48769 (local address 127.2.208.190:45669)
0620 19:01:03.077806 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:03.077811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:03.077936 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:01:03.078100 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:03.078103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:03.078154 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:03.078241 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:03.078746 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.079502 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.080368 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.080638 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.080755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.083850 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:48769: BlockingRecv error: recv got EOF from 127.2.208.130:48769 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:03.266417 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:03.261268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:57007)
0620 19:01:03.261552 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:03.261565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:03.261662 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:03.261894 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:03.261896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:03.262066 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:03.262167 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.262172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.262854 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.262856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.263446 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.263454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.263902 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.263905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.263961 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.264427 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:03.264438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:03.266307 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260620 19:01:03.266582 12095 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:03.261398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:57007 (local address 127.2.208.190:45669)
0620 19:01:03.261548 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:03.261552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:03.261658 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:01:03.261731 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:03.261733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:03.261783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:03.261861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:03.262264 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.262766 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.263552 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.263801 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.263904 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.266446 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:57007: BlockingRecv error: recv got EOF from 127.2.208.131:57007 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:03 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:03.898694 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:03.890260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:48035)
0620 19:01:03.890536 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:03.890550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:03.890650 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:03.890897 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:03.890899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:03.891074 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:03.891167 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.891172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.891954 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.891957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.892529 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:03.892537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.892979 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.892982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:03.893049 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.896584 (+ 3535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:03.896596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:03.898523 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260620 19:01:03.898762 12099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:03.890308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:48035 (local address 127.2.208.190:45669)
0620 19:01:03.890560 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:03.890564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:03.890617 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:01:03.890733 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:03.890735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:03.890777 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:03.890855 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:03.891266 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.891855 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.892629 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:03.892839 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:03.892911 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:03.898644 (+ 5733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:48035: BlockingRecv error: recv got EOF from 127.2.208.129:48035 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:04.089701 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:04.084446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:47097)
0620 19:01:04.084730 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:04.084745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:04.084849 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:04.085077 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:04.085079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:04.085250 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:04.085356 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.085362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.086054 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.086056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.086617 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.086625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.087026 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.087029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.087092 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.087630 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:04.087642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:04.089507 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
W20260620 19:01:04.089730 12099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:04.084527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:47097 (local address 127.2.208.190:45669)
0620 19:01:04.084685 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:04.084690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:04.084815 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:01:04.084930 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:04.084932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:04.084976 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:04.085041 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:04.085450 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.085958 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.086715 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.086925 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.086998 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.089628 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:47097: BlockingRecv error: recv got EOF from 127.2.208.130:47097 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:04.272478 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:04.267231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:54433)
0620 19:01:04.267521 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:04.267538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:04.267661 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:04.267908 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:04.267910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:04.268080 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:04.268172 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.268177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.268892 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.268913 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.269456 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.269463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.269844 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.269847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.269908 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.270368 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:04.270379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:04.272300 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
W20260620 19:01:04.272523 12099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:04.267315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:54433 (local address 127.2.208.190:45669)
0620 19:01:04.267463 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:04.267467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:04.267628 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:01:04.267759 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:04.267761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:04.267807 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:04.267870 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:04.268279 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.268799 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.269548 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.269755 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.269828 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.272420 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:54433: BlockingRecv error: recv got EOF from 127.2.208.131:54433 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:04 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:04.905004 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:04.899633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:37699)
0620 19:01:04.899909 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:04.899922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:04.900007 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:04.900251 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:04.900253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:04.900427 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:04.900520 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.900525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.901221 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.901223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.901762 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:04.901769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.902180 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.902183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:04.902244 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.902790 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:04.902805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:04.904832 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
W20260620 19:01:04.905084 12103 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:04.899708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37699 (local address 127.2.208.190:45669)
0620 19:01:04.899939 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:04.899942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:04.899971 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:01:04.900093 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:04.900094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:04.900136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:04.900215 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:04.900616 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.901132 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.901865 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:04.902074 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:04.902144 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:04.904949 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:37699: BlockingRecv error: recv got EOF from 127.2.208.129:37699 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:05.095695 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:05.090478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:60667)
0620 19:01:05.090741 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:05.090755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:05.090840 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:05.091073 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:05.091075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:05.091256 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:05.091394 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.091400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.092105 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.092108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.092663 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.092670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.093069 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.093072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.093133 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.093620 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:05.093631 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:05.095577 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
W20260620 19:01:05.095873 12103 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:05.090620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:60667 (local address 127.2.208.190:45669)
0620 19:01:05.090780 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:05.090784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:05.090799 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:01:05.090925 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:05.090927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:05.090969 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:05.091037 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:05.091497 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.092012 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.092758 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.092973 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.093048 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.095712 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:60667: BlockingRecv error: recv got EOF from 127.2.208.130:60667 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:05.279102 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:05.273326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:60239)
0620 19:01:05.273691 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:05.273705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:05.273793 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:05.274040 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:05.274042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:05.274212 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:05.274303 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.274308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.275013 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.275015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.275699 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.275709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.276437 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.276443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.276547 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.277139 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:05.277151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:05.278978 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":95,"threads_started":1}
W20260620 19:01:05.279254 12103 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:05.273533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:60239 (local address 127.2.208.190:45669)
0620 19:01:05.273699 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:05.273703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:05.273755 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:01:05.273881 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:05.273883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:05.273928 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:05.274005 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:05.274399 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.274916 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.275868 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.276322 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.276434 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.279126 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:60239: BlockingRecv error: recv got EOF from 127.2.208.131:60239 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:05 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:05.910888 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:05.905807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:47683)
0620 19:01:05.906080 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:05.906092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:05.906188 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:05.906425 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:05.906427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:05.906601 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:05.906694 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.906699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.907467 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.907470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.908026 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:05.908033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.908421 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.908423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:05.908479 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.908957 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:05.908968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:05.910758 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260620 19:01:05.911031 12107 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:05.905846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:47683 (local address 127.2.208.190:45669)
0620 19:01:05.906082 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:05.906086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:05.906160 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:01:05.906264 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:05.906267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:05.906309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:05.906386 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:05.906788 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.907315 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.908121 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:05.908329 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:05.908401 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:05.910890 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:47683: BlockingRecv error: recv got EOF from 127.2.208.129:47683 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:06.102257 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:06.096546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:35131)
0620 19:01:06.096843 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:06.096859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:06.096985 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:06.097286 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:06.097289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:06.097530 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:06.097657 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.097664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.098542 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.098545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.099089 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.099096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.099519 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.099522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.099581 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.100122 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:06.100136 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:06.102085 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
W20260620 19:01:06.102321 12107 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:06.096623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:35131 (local address 127.2.208.190:45669)
0620 19:01:06.096822 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:06.096828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:06.096940 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:01:06.097090 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:06.097093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:06.097150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:06.097265 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:06.097777 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.098447 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.099186 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.099430 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.099510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.102203 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:35131: BlockingRecv error: recv got EOF from 127.2.208.130:35131 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:06.285576 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:06.279891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:34099)
0620 19:01:06.280184 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:06.280197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:06.280297 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:06.280519 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:06.280522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:06.280694 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:06.280785 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.280789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.281503 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.281505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.282083 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.282090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.282568 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.282571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.282729 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.283443 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:06.283457 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:06.285463 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
W20260620 19:01:06.285794 12107 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:06.279977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:34099 (local address 127.2.208.190:45669)
0620 19:01:06.280091 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:06.280094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:06.280268 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:01:06.280367 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:06.280369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:06.280413 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:06.280484 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:06.280902 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.281411 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.282186 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.282416 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.282494 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.285603 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:34099: BlockingRecv error: recv got EOF from 127.2.208.131:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:06 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:06.916780 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:06.911702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:56627)
0620 19:01:06.911988 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:06.912002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:06.912086 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:06.912329 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:06.912332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:06.912509 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:06.912600 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.912605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.913318 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.913320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.913856 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:06.913863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.914273 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.914276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:06.914333 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.914817 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:06.914828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:06.916657 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
W20260620 19:01:06.916972 12111 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:06.911799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:56627 (local address 127.2.208.190:45669)
0620 19:01:06.912026 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:06.912029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:06.912044 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:01:06.912171 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:06.912173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:06.912215 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:06.912301 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:06.912700 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.913227 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.913959 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:06.914177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:06.914259 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:06.916797 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:56627: BlockingRecv error: recv got EOF from 127.2.208.129:56627 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:07.108883 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:07.103179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:49241)
0620 19:01:07.103546 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:07.103559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:07.103663 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:07.103947 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:07.103950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:07.104157 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:07.104274 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.104280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.105047 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.105053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.105622 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.105630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.106105 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.106108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.106205 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.106784 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:07.106796 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:07.108676 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
W20260620 19:01:07.108918 12111 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:07.103270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:49241 (local address 127.2.208.190:45669)
0620 19:01:07.103474 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:07.103479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:07.103643 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:01:07.103738 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:07.103743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:07.103809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:07.103919 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:07.104396 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.104935 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.105745 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.105992 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.106102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.108818 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:49241: BlockingRecv error: recv got EOF from 127.2.208.130:49241 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:07.291785 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:07.286472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:45321)
0620 19:01:07.286744 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:07.286757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:07.286853 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:07.287095 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:07.287097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:07.287285 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:07.287396 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.287402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.288150 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.288153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.288802 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.288811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.289274 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.289277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.289345 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.289871 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:07.289883 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:07.291667 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260620 19:01:07.291944 12111 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:07.286564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:45321 (local address 127.2.208.190:45669)
0620 19:01:07.286699 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:07.286702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:07.286828 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:01:07.286946 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:07.286948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:07.286991 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:07.287062 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:07.287505 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.288041 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.288933 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.289156 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.289276 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.291786 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:45321: BlockingRecv error: recv got EOF from 127.2.208.131:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:07 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:07.923462 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:07.917776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:39531)
0620 19:01:07.918092 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:07.918106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:07.918235 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:07.918552 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:07.918555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:07.918864 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:07.918999 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.919011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.919818 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.919821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.920377 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:07.920385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.920787 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.920790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:07.920849 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.921418 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:07.921432 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:07.923287 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
W20260620 19:01:07.923604 12115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:07.917869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:39531 (local address 127.2.208.190:45669)
0620 19:01:07.918179 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:07.918183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:07.918195 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:01:07.918348 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:07.918351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:07.918407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:07.918516 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:07.919180 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.919718 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.920476 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:07.920684 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:07.920762 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:07.923474 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:39531: BlockingRecv error: recv got EOF from 127.2.208.129:39531 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:08.115159 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:08.109606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:59305)
0620 19:01:08.109901 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:08.109915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:08.110042 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:08.110293 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:08.110296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:08.110499 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:08.110589 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.110594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.111352 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.111380 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.111954 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.111962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.112427 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.112430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.112513 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.113108 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:08.113120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:08.114996 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
W20260620 19:01:08.115290 12115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:08.109898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59305 (local address 127.2.208.190:45669)
0620 19:01:08.110050 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:08.110055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:08.110071 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:01:08.110120 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:08.110125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:08.110177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:08.110270 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:08.110708 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.111235 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.112078 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.112303 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.112422 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.115158 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:59305: BlockingRecv error: recv got EOF from 127.2.208.130:59305 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:08.297469 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:08.292536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:54861)
0620 19:01:08.292811 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:08.292825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:08.292921 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:08.293138 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:08.293140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:08.293310 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:08.293399 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.293404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.294083 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.294085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.294619 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.294626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.295001 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.295004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.295062 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.295596 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:08.295607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:08.297350 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
W20260620 19:01:08.297654 12115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:08.292598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:54861 (local address 127.2.208.190:45669)
0620 19:01:08.292727 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:08.292732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:08.292888 (+ 156us) server_negotiation.cc:407] Connection header received
0620 19:01:08.292993 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:08.292995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:08.293040 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:08.293104 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:08.293483 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.293997 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.294708 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.294910 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.294982 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.297496 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:54861: BlockingRecv error: recv got EOF from 127.2.208.131:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:08 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:08.935181 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:08.924219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:38071)
0620 19:01:08.924489 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:08.924503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:08.924588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:08.924910 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:08.924913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:08.925093 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:08.925190 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.925195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.926031 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.926034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.926616 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:08.926626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.927060 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.927063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:08.927125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.932868 (+ 5743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:08.932881 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:08.935062 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260620 19:01:08.935338 12119 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:08.924396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:38071 (local address 127.2.208.190:45669)
0620 19:01:08.924680 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:08.924683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:08.924694 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:01:08.924727 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:08.924729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:08.924767 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:08.924864 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:08.925340 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.925897 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.926736 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:08.926959 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:08.927040 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:08.935192 (+ 8152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:38071: BlockingRecv error: recv got EOF from 127.2.208.129:38071 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:09.121403 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:09.115954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:33045)
0620 19:01:09.116211 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:09.116224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:09.116330 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:09.116561 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:09.116563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:09.116735 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:09.116833 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.116839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.117525 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.117527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.118073 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.118080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.118484 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.118486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.118543 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.119029 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:09.119040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:09.121286 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
W20260620 19:01:09.121528 12119 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:09.116043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:33045 (local address 127.2.208.190:45669)
0620 19:01:09.116173 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:09.116177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:09.116289 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:01:09.116410 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:09.116412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:09.116454 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:09.116526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:09.116928 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.117435 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.118172 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.118381 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.118466 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.121415 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:33045: BlockingRecv error: recv got EOF from 127.2.208.130:33045 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.131@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:09.303643 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:09.298186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.131:55555)
0620 19:01:09.298506 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:09.298524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:09.298651 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:09.298974 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:09.298977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:09.299178 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:09.299264 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.299269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.300061 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.300067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.300623 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.300631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.301074 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.301077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.301157 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.301630 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:09.301641 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:09.303529 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
W20260620 19:01:09.303782 12119 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:09.298291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:55555 (local address 127.2.208.190:45669)
0620 19:01:09.298461 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:09.298466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:09.298599 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:01:09.298767 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:09.298771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:09.298840 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:09.298948 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:09.299423 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.299952 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.300739 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.300954 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.301072 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.303665 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.131:55555: BlockingRecv error: recv got EOF from 127.2.208.131:55555 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:09 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.129@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:09.941906 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:09.936217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.129:48609)
0620 19:01:09.936512 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:09.936528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:09.936637 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:09.936909 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:09.936911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:09.937088 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:09.937195 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.937200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.938006 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.938013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.938713 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:09.938724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.939384 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.939388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:09.939483 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.939957 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:09.939971 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:09.941773 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
W20260620 19:01:09.942098 12123 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:09.936304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:48609 (local address 127.2.208.190:45669)
0620 19:01:09.936549 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:09.936553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:09.936613 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:01:09.936724 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:09.936726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:09.936769 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:09.936872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:09.937297 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.937857 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.938876 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:09.939253 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:09.939389 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:09.941928 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.129:48609: BlockingRecv error: recv got EOF from 127.2.208.129:48609 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
Jun 20 19:01:10 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:01:10 dist-test-slave-tsm4 krb5kdc[11065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.208.130@KRBTEST.COM for oryx/127.2.208.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:01:10.128185 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.122553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:45669 (local address 127.2.208.130:51809)
0620 19:01:10.122885 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:10.122902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:10.123057 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:10.123339 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:10.123342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:10.123579 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:10.123673 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.123679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.124449 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.124453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.125018 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.125026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.125488 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.125492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.125578 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.126060 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:10.126072 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:10.127998 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.208.190:45669: Server oryx/127.2.208.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":84,"threads_started":1}
W20260620 19:01:10.128242 12123 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:10.122693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:51809 (local address 127.2.208.190:45669)
0620 19:01:10.122887 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:10.122893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:10.122997 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:01:10.123164 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:10.123167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:10.123223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:10.123317 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:10.123804 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.124332 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.125138 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.125375 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.125483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.128127 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.208.130:51809: BlockingRecv error: recv got EOF from 127.2.208.130:51809 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:01:10.204033 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11648
I20260620 19:01:10.209604 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11800
I20260620 19:01:10.215057 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11932
I20260620 19:01:10.220141 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 11578
2026-06-20T19:01:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18001 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:01:10Z Disabled control of system clock
I20260620 19:01:10.250958 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38859
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:01:10.360985 12128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:10.361256 12128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:10.361290 12128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:10.365756 12128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:01:10.365880 12128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:10.365953 12128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:10.365996 12128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:01:10.366022 12128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:01:10.371130 12128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:38859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:10.372460 12128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:10.373926 12128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:10.382431 12133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:10.382432 12136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.382634 12128 server_base.cc:1034] running on GCE node
W20260620 19:01:10.382448 12134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.383242 12128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:10.384373 12128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:10.385552 12128 hybrid_clock.cc:648] HybridClock initialized: now 1781982070385539 us; error 28 us; skew 500 ppm
I20260620 19:01:10.390349 12128 webserver.cc:460] Webserver started at http://127.2.208.190:43613/ using document root <none> and password file <none>
I20260620 19:01:10.390975 12128 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:10.391068 12128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:10.391297 12128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:10.393388 12128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "f67e754403bf47708e6295e81e73ab51"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "18aae77ff9ec8e8385e58d0258234ac8"
server_key_iv: "43bc52c1bb5dbdc9b2e86ece67692c8e"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.393971 12128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "f67e754403bf47708e6295e81e73ab51"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "18aae77ff9ec8e8385e58d0258234ac8"
server_key_iv: "43bc52c1bb5dbdc9b2e86ece67692c8e"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.397579 12128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:01:10.400010 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.401288 12128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:10.401424 12128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "f67e754403bf47708e6295e81e73ab51"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "18aae77ff9ec8e8385e58d0258234ac8"
server_key_iv: "43bc52c1bb5dbdc9b2e86ece67692c8e"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.401538 12128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:10.415423 12128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:10.416117 12128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:10.416308 12128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:10.422809 12128 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:38859
I20260620 19:01:10.422832 12194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:38859 every 8 connection(s)
I20260620 19:01:10.423732 12128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:01:10.426560 12195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:10.426891 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12128
I20260620 19:01:10.427013 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:01:10.427289 2882 external_mini_cluster.cc:1428] Setting key 3280cd55d3c6a4a9afcfa728720960e2
I20260620 19:01:10.433092 12195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51: Bootstrap starting.
I20260620 19:01:10.435252 12195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:10.436117 12195 log.cc:824] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:10.436398 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.428698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.2.208.190:38859)
0620 19:01:10.429337 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:10.429351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:10.429391 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:01:10.430038 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:10.430055 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:10.430444 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:10.430775 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:10.431195 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.432206 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.433104 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.433628 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.433808 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.434662 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:10.434668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:10.434681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:10.435067 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:10.435120 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:10.435138 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:10.435240 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:10.435427 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:01:10.435670 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
I20260620 19:01:10.438160 12195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51: No bootstrap required, opened a new log
I20260620 19:01:10.441694 12195 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } }
I20260620 19:01:10.441919 12195 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:10.441987 12195 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67e754403bf47708e6295e81e73ab51, State: Initialized, Role: FOLLOWER
I20260620 19:01:10.442391 12195 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } }
I20260620 19:01:10.442526 12195 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:01:10.442587 12195 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:01:10.442706 12195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:10.443802 12195 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } }
I20260620 19:01:10.444142 12195 leader_election.cc:302] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67e754403bf47708e6295e81e73ab51; no voters:
I20260620 19:01:10.444453 12195 leader_election.cc:288] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:01:10.444598 12200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:10.444859 12200 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [term 1 LEADER]: Becoming Leader. State: Replica: f67e754403bf47708e6295e81e73ab51, State: Running, Role: LEADER
I20260620 19:01:10.445139 12200 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } }
I20260620 19:01:10.445623 12195 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:01:10.446764 12202 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67e754403bf47708e6295e81e73ab51. Latest consensus state: current_term: 1 leader_uuid: "f67e754403bf47708e6295e81e73ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } } }
I20260620 19:01:10.446893 12202 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:10.447105 12201 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67e754403bf47708e6295e81e73ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67e754403bf47708e6295e81e73ab51" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38859 } } }
I20260620 19:01:10.447212 12201 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:10.447513 12209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:01:10.450229 12209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:01:10.455585 12209 catalog_manager.cc:1347] Generated new cluster ID: 400464b1905342608b3a09dd1167956d
I20260620 19:01:10.455650 12209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:01:10.462664 12209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:01:10.464061 12209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:01:10.479185 12209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67e754403bf47708e6295e81e73ab51: Generated new TSK 0
I20260620 19:01:10.480113 12209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:01:10.486536 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--builtin_ntp_servers=127.2.208.148:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:01:10.615969 12219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:10.616250 12219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:10.616321 12219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:10.620411 12219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:10.620496 12219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:10.620589 12219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:01:10.625336 12219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:10.626461 12219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:10.627899 12219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:10.637140 12224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.637280 12219 server_base.cc:1034] running on GCE node
W20260620 19:01:10.637136 12227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:10.637151 12225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.637804 12219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:10.638412 12219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:10.639590 12219 hybrid_clock.cc:648] HybridClock initialized: now 1781982070639570 us; error 31 us; skew 500 ppm
I20260620 19:01:10.641335 12219 webserver.cc:460] Webserver started at http://127.2.208.129:42661/ using document root <none> and password file <none>
I20260620 19:01:10.641914 12219 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:10.642000 12219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:10.642225 12219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:10.644342 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "8f5886e1dfcd4c3cb63e0f7fafbfc5a3"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "6d8694e60623d92d808b47f1f780ddd7"
server_key_iv: "a37fdab2887c0f9d6bdba7a50c9fdaf4"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.644902 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5886e1dfcd4c3cb63e0f7fafbfc5a3"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "6d8694e60623d92d808b47f1f780ddd7"
server_key_iv: "a37fdab2887c0f9d6bdba7a50c9fdaf4"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.649315 12219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:01:10.651906 12233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.653036 12219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:10.653177 12219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "8f5886e1dfcd4c3cb63e0f7fafbfc5a3"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "6d8694e60623d92d808b47f1f780ddd7"
server_key_iv: "a37fdab2887c0f9d6bdba7a50c9fdaf4"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.653290 12219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:10.677582 12219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:10.678301 12219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:10.678507 12219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:10.679095 12219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:10.680115 12219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:10.680186 12219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.680259 12219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:10.680302 12219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.689249 12219 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:36535
I20260620 19:01:10.689273 12346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:36535 every 8 connection(s)
I20260620 19:01:10.690138 12219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:01:10.693581 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12219
I20260620 19:01:10.693711 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:01:10.693985 2882 external_mini_cluster.cc:1428] Setting key 47acbecc2c09f307aaa16ddbddaaf7fd
I20260620 19:01:10.695988 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--builtin_ntp_servers=127.2.208.148:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:01:10.699735 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.691931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:38701 (local address 127.2.208.190:38859)
0620 19:01:10.692105 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:10.692110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:10.693154 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 19:01:10.694708 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:10.694712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:10.694767 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:10.694873 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:10.695864 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.696697 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.697802 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.698037 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.698150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.699213 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:10.699216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:10.699219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:10.699239 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:10.699258 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:10.699265 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:10.699350 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:10.699528 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:01:10.699593 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73}
I20260620 19:01:10.700546 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.692347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38859 (local address 127.2.208.129:38701)
0620 19:01:10.693004 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:10.693046 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:10.694513 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:10.695018 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:10.695025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:10.695102 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:10.695714 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.695728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.696827 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.696829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.697677 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.697686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.698185 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.698188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.698348 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.698960 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:10.698962 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:10.699001 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:10.699061 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:10.699352 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:10.699382 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:10.699391 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:01:10.699527 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 19:01:10.699792 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260620 19:01:10.701889 12347 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38859
I20260620 19:01:10.702111 12347 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:10.702602 12347 heartbeater.cc:511] Master 127.2.208.190:38859 requested a full tablet report, sending...
I20260620 19:01:10.704365 12159 ts_manager.cc:194] Registered new tserver with Master: 8f5886e1dfcd4c3cb63e0f7fafbfc5a3 (127.2.208.129:36535)
I20260620 19:01:10.706135 12159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.129:38701
W20260620 19:01:10.808044 12350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:10.808300 12350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:10.808336 12350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:10.812379 12350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:10.812429 12350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:10.812481 12350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:01:10.817103 12350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:10.818116 12350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:10.819437 12350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:10.828972 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:10.828971 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.829198 12350 server_base.cc:1034] running on GCE node
W20260620 19:01:10.828982 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:10.829655 12350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:10.830296 12350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:10.831447 12350 hybrid_clock.cc:648] HybridClock initialized: now 1781982070831407 us; error 46 us; skew 500 ppm
I20260620 19:01:10.833345 12350 webserver.cc:460] Webserver started at http://127.2.208.130:34105/ using document root <none> and password file <none>
I20260620 19:01:10.834014 12350 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:10.834080 12350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:10.834246 12350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:10.836337 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "806c5373dc6946ad837b7549d521b38a"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "04456749fd45d5b5a63030c2967c25d3"
server_key_iv: "1fe2e3e6ced7ebea6dae488654cf9a3f"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.836849 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "806c5373dc6946ad837b7549d521b38a"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "04456749fd45d5b5a63030c2967c25d3"
server_key_iv: "1fe2e3e6ced7ebea6dae488654cf9a3f"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.840505 12350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 19:01:10.843036 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.844313 12350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:01:10.844440 12350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "806c5373dc6946ad837b7549d521b38a"
format_stamp: "Formatted at 2026-06-20 19:01:10 on dist-test-slave-tsm4"
server_key: "04456749fd45d5b5a63030c2967c25d3"
server_key_iv: "1fe2e3e6ced7ebea6dae488654cf9a3f"
server_key_version: "encryptionkey@0"
I20260620 19:01:10.844552 12350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:10.865530 12350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:10.866283 12350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:10.866485 12350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:10.867089 12350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:10.868113 12350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:10.868182 12350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.868249 12350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:10.868300 12350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:10.877200 12350 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:35637
I20260620 19:01:10.877220 12477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:35637 every 8 connection(s)
I20260620 19:01:10.878135 12350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:01:10.882285 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12350
I20260620 19:01:10.882421 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:01:10.882746 2882 external_mini_cluster.cc:1428] Setting key 2e6f4d63d76fff9f8c1a1ae8bc560ff9
I20260620 19:01:10.884868 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--builtin_ntp_servers=127.2.208.148:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:01:10.888111 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.879968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59181 (local address 127.2.208.190:38859)
0620 19:01:10.880113 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:10.880116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:10.881174 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 19:01:10.882835 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:10.882839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:10.882906 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:10.882988 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:10.884150 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.884913 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.886147 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.886370 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.886478 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.887659 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:10.887662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:10.887664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:10.887676 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:10.887694 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:10.887701 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:10.887761 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:10.887959 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:01:10.888004 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:01:10.889423 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:10.880369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38859 (local address 127.2.208.130:59181)
0620 19:01:10.881020 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:10.881062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:10.882545 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:10.883213 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:10.883220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:10.883308 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:10.884000 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.884021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.885135 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.885138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.886019 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:10.886028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:10.886520 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:10.886525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:10.886706 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:10.887339 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:10.887342 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:10.887423 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:10.887508 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:10.887785 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:10.887796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:10.887806 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 19:01:10.887947 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 19:01:10.888249 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260620 19:01:10.890637 12478 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38859
I20260620 19:01:10.890868 12478 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:10.891422 12478 heartbeater.cc:511] Master 127.2.208.190:38859 requested a full tablet report, sending...
I20260620 19:01:10.892432 12159 ts_manager.cc:194] Registered new tserver with Master: 806c5373dc6946ad837b7549d521b38a (127.2.208.130:35637)
I20260620 19:01:10.892976 12159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.130:59181
W20260620 19:01:10.997709 12481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:10.998014 12481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:10.998091 12481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:11.002146 12481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:11.002225 12481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:11.002310 12481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:01:11.007249 12481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:11.008387 12481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:11.009817 12481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:11.019163 12486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.019183 12489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.019171 12487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:11.019207 12481 server_base.cc:1034] running on GCE node
I20260620 19:01:11.019840 12481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:11.020473 12481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:11.021659 12481 hybrid_clock.cc:648] HybridClock initialized: now 1781982071021618 us; error 63 us; skew 500 ppm
I20260620 19:01:11.023635 12481 webserver.cc:460] Webserver started at http://127.2.208.131:42853/ using document root <none> and password file <none>
I20260620 19:01:11.024225 12481 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:11.024319 12481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:11.024554 12481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:11.026753 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "798a9b1825c64436a93bd043b3968af9"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "6a99bab8e8363e940c74bb7bd759f2b0"
server_key_iv: "99c5719bfd841c42a955516b149b3421"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.027320 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "798a9b1825c64436a93bd043b3968af9"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "6a99bab8e8363e940c74bb7bd759f2b0"
server_key_iv: "99c5719bfd841c42a955516b149b3421"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.031152 12481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:01:11.033780 12495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.035208 12481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:01:11.035315 12481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "798a9b1825c64436a93bd043b3968af9"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "6a99bab8e8363e940c74bb7bd759f2b0"
server_key_iv: "99c5719bfd841c42a955516b149b3421"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.035491 12481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:11.056524 12481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:11.057183 12481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:11.057343 12481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:11.057946 12481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:11.058892 12481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:11.058939 12481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.059007 12481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:11.059041 12481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.067770 12481 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:40365
I20260620 19:01:11.067791 12608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:40365 every 8 connection(s)
I20260620 19:01:11.068702 12481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:01:11.071635 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12481
I20260620 19:01:11.071741 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:01:11.072063 2882 external_mini_cluster.cc:1428] Setting key 40b39092c21c14be265e9151fd73d89a
I20260620 19:01:11.077881 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.070440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:38719 (local address 127.2.208.190:38859)
0620 19:01:11.070609 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.070614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.071713 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 19:01:11.073580 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.073582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.073626 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.073696 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.074681 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:11.075203 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:11.076267 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:11.076471 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:11.076548 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:11.077482 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:11.077485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:11.077487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:11.077498 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:11.077515 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:11.077523 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:11.077578 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:11.077705 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:01:11.077762 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:01:11.078751 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.070859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:38859 (local address 127.2.208.131:38719)
0620 19:01:11.071568 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:11.071607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:11.073345 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:11.073896 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:11.073905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:11.074019 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:11.074567 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:11.074580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:11.075300 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:11.075302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:11.076170 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:11.076179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:11.076589 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:11.076592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:11.076724 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:11.077253 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:11.077255 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:11.077291 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:11.077350 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:11.077590 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:11.077596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:11.077600 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:01:11.077690 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:01:11.077899 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260620 19:01:11.079943 12609 heartbeater.cc:347] Connected to a master server at 127.2.208.190:38859
I20260620 19:01:11.080168 12609 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:11.080658 12609 heartbeater.cc:511] Master 127.2.208.190:38859 requested a full tablet report, sending...
I20260620 19:01:11.081610 12159 ts_manager.cc:194] Registered new tserver with Master: 798a9b1825c64436a93bd043b3968af9 (127.2.208.131:40365)
I20260620 19:01:11.082171 12159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.131:38719
I20260620 19:01:11.085695 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:01:11.088510 12198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:11.087536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.2.208.190:38859)
0620 19:01:11.087664 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.087670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.087877 (+ 207us) server_negotiation.cc:407] Connection header received
0620 19:01:11.088024 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.088027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.088074 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.088156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.088390 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39434: BlockingRecv error: recv got EOF from 127.0.0.1:39434 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:01:11.088784 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12219
I20260620 19:01:11.094331 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12350
I20260620 19:01:11.099582 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12481
I20260620 19:01:11.104671 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12128
2026-06-20T19:01:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (882 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:01:11Z Disabled control of system clock
I20260620 19:01:11.132972 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44727
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:01:11.243532 12620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:11.243799 12620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:11.243837 12620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:11.247839 12620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:01:11.247892 12620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:11.247907 12620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:11.247924 12620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:01:11.247941 12620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:01:11.252661 12620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:11.253685 12620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:11.255023 12620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:11.263720 12628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.263741 12626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.263726 12625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:11.264925 12620 server_base.cc:1034] running on GCE node
I20260620 19:01:11.265403 12620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:11.266498 12620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:11.267652 12620 hybrid_clock.cc:648] HybridClock initialized: now 1781982071267629 us; error 33 us; skew 500 ppm
I20260620 19:01:11.269526 12620 webserver.cc:460] Webserver started at http://127.2.208.190:41585/ using document root <none> and password file <none>
I20260620 19:01:11.270102 12620 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:11.270164 12620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:11.270418 12620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:11.272658 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "3d4a09679e9d43c8874b50ecf9df8d53"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "21c0d6b147c434649176a2843ecd48b3"
server_key_iv: "d9b8e7eb69e0cef196534693022eb861"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.273238 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "3d4a09679e9d43c8874b50ecf9df8d53"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "21c0d6b147c434649176a2843ecd48b3"
server_key_iv: "d9b8e7eb69e0cef196534693022eb861"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.276775 12620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:01:11.279054 12634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.280292 12620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:11.280427 12620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "3d4a09679e9d43c8874b50ecf9df8d53"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "21c0d6b147c434649176a2843ecd48b3"
server_key_iv: "d9b8e7eb69e0cef196534693022eb861"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.280539 12620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:11.285202 12620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:11.285827 12620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:11.286008 12620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:11.292824 12620 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44727
I20260620 19:01:11.292821 12686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44727 every 8 connection(s)
I20260620 19:01:11.293823 12620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:01:11.297082 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:11.298429 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12620
I20260620 19:01:11.298538 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:01:11.298839 2882 external_mini_cluster.cc:1428] Setting key 0beafc9b6dee1e4ebb5c88ae14e76299
I20260620 19:01:11.303515 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53: Bootstrap starting.
I20260620 19:01:11.304312 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.300270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.2.208.190:44727)
0620 19:01:11.300865 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.300880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.300915 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:01:11.301563 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.301578 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.301911 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.302180 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.303001 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:11.303005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:11.303017 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:11.303026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:11.303078 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:11.303094 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:11.303201 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:11.303305 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:01:11.303614 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":133,"threads_started":1}
I20260620 19:01:11.305684 12687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:11.306471 12687 log.cc:824] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:11.308297 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53: No bootstrap required, opened a new log
I20260620 19:01:11.311909 12687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } }
I20260620 19:01:11.312125 12687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:11.312199 12687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4a09679e9d43c8874b50ecf9df8d53, State: Initialized, Role: FOLLOWER
I20260620 19:01:11.312594 12687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } }
I20260620 19:01:11.312732 12687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:01:11.312793 12687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:01:11.312912 12687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:11.313786 12687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } }
I20260620 19:01:11.314153 12687 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4a09679e9d43c8874b50ecf9df8d53; no voters:
I20260620 19:01:11.314503 12687 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:01:11.314618 12692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:11.314865 12692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [term 1 LEADER]: Becoming Leader. State: Replica: 3d4a09679e9d43c8874b50ecf9df8d53, State: Running, Role: LEADER
I20260620 19:01:11.315198 12692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } }
I20260620 19:01:11.315838 12687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:01:11.317065 12692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4a09679e9d43c8874b50ecf9df8d53. Latest consensus state: current_term: 1 leader_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } } }
I20260620 19:01:11.317190 12692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:11.317440 12694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4a09679e9d43c8874b50ecf9df8d53" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44727 } } }
I20260620 19:01:11.317543 12694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:11.318229 12701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:01:11.321455 12701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:01:11.326509 12701 catalog_manager.cc:1347] Generated new cluster ID: 7803546555904c9da998d389c12008c6
I20260620 19:01:11.326582 12701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:01:11.335752 12701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:01:11.336071 12701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:01:11.343032 12701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d4a09679e9d43c8874b50ecf9df8d53: Generated new TSK 0
I20260620 19:01:11.343861 12701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:01:11.352401 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:01:11.462394 12711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:11.462623 12711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:11.462663 12711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:11.466656 12711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:11.466714 12711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:11.466825 12711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:01:11.471666 12711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:11.472721 12711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:11.474117 12711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:11.483253 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.483284 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.483248 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:11.483521 12711 server_base.cc:1034] running on GCE node
I20260620 19:01:11.483951 12711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:11.484652 12711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:11.485836 12711 hybrid_clock.cc:648] HybridClock initialized: now 1781982071485817 us; error 29 us; skew 500 ppm
I20260620 19:01:11.487717 12711 webserver.cc:460] Webserver started at http://127.2.208.129:36687/ using document root <none> and password file <none>
I20260620 19:01:11.488310 12711 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:11.488401 12711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:11.488626 12711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:11.490597 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "c7f2e56bd7d546e294b9272cd179da03"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "98136492b1d58e7a599a3c9b1152b4ed"
server_key_iv: "77ec0727a87b9a3ba39fdb09d85d9014"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.491138 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "c7f2e56bd7d546e294b9272cd179da03"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "98136492b1d58e7a599a3c9b1152b4ed"
server_key_iv: "77ec0727a87b9a3ba39fdb09d85d9014"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.494608 12711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:01:11.497052 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.498217 12711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:01:11.498349 12711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "c7f2e56bd7d546e294b9272cd179da03"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "98136492b1d58e7a599a3c9b1152b4ed"
server_key_iv: "77ec0727a87b9a3ba39fdb09d85d9014"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.498462 12711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:11.502907 12711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:11.503544 12711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:11.503721 12711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:11.504245 12711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:11.505112 12711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:11.505182 12711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.505249 12711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:11.505308 12711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.513602 12711 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:37653
I20260620 19:01:11.513628 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:37653 every 8 connection(s)
I20260620 19:01:11.514469 12711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:01:11.518536 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12711
I20260620 19:01:11.518666 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:01:11.518986 2882 external_mini_cluster.cc:1428] Setting key b2394eb89bffa45073b016b13b789ec7
I20260620 19:01:11.520828 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.516300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:58835 (local address 127.2.208.190:44727)
0620 19:01:11.516436 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.516440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.517611 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 19:01:11.519145 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.519148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.519184 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.519248 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.520408 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:11.520412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:11.520415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:11.520426 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:11.520452 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:11.520460 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:11.520529 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:11.520691 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:01:11.520725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:01:11.521009 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:01:11.521932 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.516715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44727 (local address 127.2.208.129:58835)
0620 19:01:11.517498 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:11.517536 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:11.518893 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:11.519448 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:11.519463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:11.519549 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:11.520145 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:11.520146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:11.520185 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:11.520248 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:11.520544 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:11.520552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:11.520558 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:01:11.520727 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 19:01:11.520964 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260620 19:01:11.523058 12839 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44727
I20260620 19:01:11.523304 12839 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:11.523763 12839 heartbeater.cc:511] Master 127.2.208.190:44727 requested a full tablet report, sending...
I20260620 19:01:11.525233 12651 ts_manager.cc:194] Registered new tserver with Master: c7f2e56bd7d546e294b9272cd179da03 (127.2.208.129:37653)
W20260620 19:01:11.629122 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:11.629410 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:11.629496 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:11.633522 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:11.633594 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:11.633682 12842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:01:11.638392 12842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:11.639487 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:11.640818 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:11.649961 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.650225 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.649957 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:11.650274 12842 server_base.cc:1034] running on GCE node
I20260620 19:01:11.650769 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:11.651597 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:11.652786 12842 hybrid_clock.cc:648] HybridClock initialized: now 1781982071652768 us; error 34 us; skew 500 ppm
I20260620 19:01:11.654623 12842 webserver.cc:460] Webserver started at http://127.2.208.130:44581/ using document root <none> and password file <none>
I20260620 19:01:11.655140 12842 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:11.655198 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:11.655418 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:11.657392 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "79ccc9fdfbfc4bfe87019d3a93aee855"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "32b9c7baae3c25416cb68254fb67c0a7"
server_key_iv: "243506be6b83725e91a5414b43117773"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.657913 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "79ccc9fdfbfc4bfe87019d3a93aee855"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "32b9c7baae3c25416cb68254fb67c0a7"
server_key_iv: "243506be6b83725e91a5414b43117773"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.661386 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:01:11.663796 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.664930 12842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:11.665014 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "79ccc9fdfbfc4bfe87019d3a93aee855"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "32b9c7baae3c25416cb68254fb67c0a7"
server_key_iv: "243506be6b83725e91a5414b43117773"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.665097 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:11.669387 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:11.669924 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:11.670097 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:11.670612 12842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:11.671520 12842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:11.671566 12842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.671626 12842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:11.671660 12842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.680455 12842 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:42371
I20260620 19:01:11.680543 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:42371 every 8 connection(s)
I20260620 19:01:11.681327 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:01:11.687315 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.683048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:57459 (local address 127.2.208.190:44727)
0620 19:01:11.683176 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.683180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.684251 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 19:01:11.685695 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.685699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.685744 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.685831 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.686824 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:11.686827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:11.686830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:11.686841 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:11.686861 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:11.686876 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:11.686969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:11.687111 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:01:11.687155 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:01:11.687590 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12842
I20260620 19:01:11.687745 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:01:11.688083 2882 external_mini_cluster.cc:1428] Setting key 1893ed9084160f6b469ca87ed14dea8d
I20260620 19:01:11.688298 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.683426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44727 (local address 127.2.208.130:57459)
0620 19:01:11.684090 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:11.684130 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:11.685489 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:11.685991 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:11.685998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:11.686084 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:11.686581 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:11.686582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:11.686619 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:11.686684 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:11.686962 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:11.686970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:11.686977 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:01:11.687153 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 19:01:11.687426 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260620 19:01:11.689385 12970 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44727
I20260620 19:01:11.689600 12970 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:11.689890 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--builtin_ntp_servers=127.2.208.148:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:01:11.690040 12970 heartbeater.cc:511] Master 127.2.208.190:44727 requested a full tablet report, sending...
I20260620 19:01:11.691078 12651 ts_manager.cc:194] Registered new tserver with Master: 79ccc9fdfbfc4bfe87019d3a93aee855 (127.2.208.130:42371)
W20260620 19:01:11.799080 12973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:11.799400 12973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:11.799468 12973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:11.803628 12973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:11.803699 12973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:11.803784 12973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:01:11.808642 12973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:11.809719 12973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:11.811069 12973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:11.820575 12981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.820631 12978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:11.820626 12979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:11.820772 12973 server_base.cc:1034] running on GCE node
I20260620 19:01:11.821350 12973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:11.822036 12973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:11.823216 12973 hybrid_clock.cc:648] HybridClock initialized: now 1781982071823209 us; error 31 us; skew 500 ppm
I20260620 19:01:11.825028 12973 webserver.cc:460] Webserver started at http://127.2.208.131:40779/ using document root <none> and password file <none>
I20260620 19:01:11.825608 12973 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:11.825699 12973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:11.825932 12973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:11.828040 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "373d95c30a554fd9881d852f609b5170"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "2bd22f766fd9d6d74677b27a080fbf4e"
server_key_iv: "2b862023f3be4e3f54acc4701439bb18"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.828591 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "373d95c30a554fd9881d852f609b5170"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "2bd22f766fd9d6d74677b27a080fbf4e"
server_key_iv: "2b862023f3be4e3f54acc4701439bb18"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.832180 12973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:01:11.834729 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.835994 12973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:11.836125 12973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "373d95c30a554fd9881d852f609b5170"
format_stamp: "Formatted at 2026-06-20 19:01:11 on dist-test-slave-tsm4"
server_key: "2bd22f766fd9d6d74677b27a080fbf4e"
server_key_iv: "2b862023f3be4e3f54acc4701439bb18"
server_key_version: "encryptionkey@0"
I20260620 19:01:11.836241 12973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:11.840771 12973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:11.841358 12973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:11.841534 12973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:11.842049 12973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:11.842934 12973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:11.843009 12973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.843096 12973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:11.843145 12973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:11.851788 12973 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:39211
I20260620 19:01:11.851806 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:39211 every 8 connection(s)
I20260620 19:01:11.852661 12973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:01:11.856143 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 12973
I20260620 19:01:11.856256 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:01:11.856571 2882 external_mini_cluster.cc:1428] Setting key 01f8055c45f3fcfd6c5d985022259564
I20260620 19:01:11.859163 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.854357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:57015 (local address 127.2.208.190:44727)
0620 19:01:11.854503 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.854508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.855664 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 19:01:11.857347 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.857350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.857396 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.857477 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.858788 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:11.858791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:11.858794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:01:11.858805 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:11.858829 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:01:11.858839 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:11.858920 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:11.859031 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:01:11.859066 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:01:11.860103 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:11.854837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44727 (local address 127.2.208.131:57015)
0620 19:01:11.855524 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:11.855564 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:11.857132 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:11.857685 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:11.857695 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:11.857818 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:11.858481 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:01:11.858483 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:11.858524 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:01:11.858615 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:11.858922 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:11.858929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:11.858933 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:01:11.859071 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 19:01:11.859346 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260620 19:01:11.861176 13101 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44727
I20260620 19:01:11.861361 13101 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:11.861696 13101 heartbeater.cc:511] Master 127.2.208.190:44727 requested a full tablet report, sending...
I20260620 19:01:11.862577 12651 ts_manager.cc:194] Registered new tserver with Master: 373d95c30a554fd9881d852f609b5170 (127.2.208.131:39211)
I20260620 19:01:11.870388 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:01:11.873037 12690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:01:11.872275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.2.208.190:44727)
0620 19:01:11.872391 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:11.872394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:11.872549 (+ 155us) server_negotiation.cc:407] Connection header received
0620 19:01:11.872644 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:11.872646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:11.872676 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:11.872742 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:11.872932 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44824: BlockingRecv error: recv got EOF from 127.0.0.1:44824 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:01:11.873212 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12711
I20260620 19:01:11.879084 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12842
I20260620 19:01:11.884492 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12973
I20260620 19:01:11.890450 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 12620
2026-06-20T19:01:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (786 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:01:11 dist-test-slave-tsm4 krb5kdc[13111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:01:11 dist-test-slave-tsm4 krb5kdc[13111](info): set up 2 sockets
Jun 20 19:01:11 dist-test-slave-tsm4 krb5kdc[13111](info): commencing operation
krb5kdc: starting...
W20260620 19:01:12.920185 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:01:13.042491 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37883
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:37883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:13.154598 13124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:13.154852 13124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:13.154891 13124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:13.159051 13124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:01:13.159106 13124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:13.159125 13124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:13.159142 13124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:01:13.159157 13124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:01:13.164583 13124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:37883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:13.165648 13124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:13.166706 13124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:13.175268 13132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.175278 13129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.175284 13130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:13.175557 13124 server_base.cc:1034] running on GCE node
I20260620 19:01:13.176235 13124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:13.177381 13124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:13.178567 13124 hybrid_clock.cc:648] HybridClock initialized: now 1781982073178557 us; error 33 us; skew 500 ppm
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:13.181226 13124 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:01:13.182128 13124 webserver.cc:460] Webserver started at http://127.2.208.190:42195/ using document root <none> and password file <none>
I20260620 19:01:13.182690 13124 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:13.182787 13124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:13.183015 13124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:13.185109 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "a030263fe1974e10bbd6cf5699a9f6b9"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "2c4626af7eb849175222b2876997cbc1"
server_key_iv: "19d8a1cc3269bc733188f3e1a7e5eec5"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.185716 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "a030263fe1974e10bbd6cf5699a9f6b9"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "2c4626af7eb849175222b2876997cbc1"
server_key_iv: "19d8a1cc3269bc733188f3e1a7e5eec5"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.189503 13124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:01:13.191912 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.193192 13124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:13.193351 13124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "a030263fe1974e10bbd6cf5699a9f6b9"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "2c4626af7eb849175222b2876997cbc1"
server_key_iv: "19d8a1cc3269bc733188f3e1a7e5eec5"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.193473 13124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:13.207820 13124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:13.210856 13124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:13.211056 13124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:13.217941 13124 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:37883
I20260620 19:01:13.218215 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:37883 every 8 connection(s)
I20260620 19:01:13.218966 13124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:01:13.221803 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:13.227206 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: Bootstrap starting.
I20260620 19:01:13.228945 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 13124
I20260620 19:01:13.229080 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:01:13.229339 2882 external_mini_cluster.cc:1428] Setting key 066c0c855492633d780898ad43bde1eb
I20260620 19:01:13.229842 13192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:13.230674 13192 log.cc:824] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:13.232695 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: No bootstrap required, opened a new log
I20260620 19:01:13.236905 13192 raft_consensus.cc:348] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } }
I20260620 19:01:13.237198 13192 raft_consensus.cc:374] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:13.237263 13192 raft_consensus.cc:729] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a030263fe1974e10bbd6cf5699a9f6b9, State: Initialized, Role: FOLLOWER
I20260620 19:01:13.237800 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } }
I20260620 19:01:13.237996 13192 raft_consensus.cc:388] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:01:13.238072 13192 raft_consensus.cc:482] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:01:13.238168 13192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:13.239128 13192 raft_consensus.cc:504] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } }
I20260620 19:01:13.239554 13192 leader_election.cc:302] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a030263fe1974e10bbd6cf5699a9f6b9; no voters:
I20260620 19:01:13.239965 13192 leader_election.cc:288] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:01:13.240314 13197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:13.240502 13197 raft_consensus.cc:686] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [term 1 LEADER]: Becoming Leader. State: Replica: a030263fe1974e10bbd6cf5699a9f6b9, State: Running, Role: LEADER
I20260620 19:01:13.240792 13197 consensus_queue.cc:237] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } }
I20260620 19:01:13.241288 13192 sys_catalog.cc:564] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:01:13.241992 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a030263fe1974e10bbd6cf5699a9f6b9. Latest consensus state: current_term: 1 leader_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } } }
I20260620 19:01:13.242769 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:13.242647 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a030263fe1974e10bbd6cf5699a9f6b9" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37883 } } }
I20260620 19:01:13.243124 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:13.244822 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.230487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.2.208.190:37883)
0620 19:01:13.230945 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:13.230960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:13.230994 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:01:13.231834 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:13.231854 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:13.232169 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:13.232442 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:13.233359 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.234345 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.235613 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.236120 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.236237 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.239105 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:13.239111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:13.239124 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:13.239151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:13.241184 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.241693 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.241702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.241711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.241856 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.242231 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.242234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.242236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.242793 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:13.243044 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:13.243534 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 19:01:13.243840 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":168,"threads_started":1}
W20260620 19:01:13.246809 13212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:01:13.246881 13212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:01:13.246951 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:01:13.247817 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:01:13.253497 13209 catalog_manager.cc:1347] Generated new cluster ID: 1db904bf9e464a728b3abcf01d6b4016
I20260620 19:01:13.253573 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:01:13.261618 13209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:01:13.262801 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:01:13.270506 13209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: Generated new TSK 0
I20260620 19:01:13.271270 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:01:13.306368 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:13.432969 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:13.433248 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:13.433293 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:13.437310 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:13.437374 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:13.437434 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:01:13.442829 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:13.443990 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:13.444989 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:13.454579 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.454681 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.454579 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:13.454948 13218 server_base.cc:1034] running on GCE node
I20260620 19:01:13.455480 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:13.456095 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:13.457257 13218 hybrid_clock.cc:648] HybridClock initialized: now 1781982073457243 us; error 26 us; skew 500 ppm
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:13.460037 13218 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:01:13.460975 13218 webserver.cc:460] Webserver started at http://127.2.208.129:38807/ using document root <none> and password file <none>
I20260620 19:01:13.461555 13218 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:13.461645 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:13.461881 13218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:13.464011 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "15d45a06de79279ead920a47d0d69b8c"
server_key_iv: "92cf2d29ceeca7fdede83b80ae3a2feb"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.464572 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "15d45a06de79279ead920a47d0d69b8c"
server_key_iv: "92cf2d29ceeca7fdede83b80ae3a2feb"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.468358 13218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:01:13.470851 13233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.472191 13218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:13.472339 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "15d45a06de79279ead920a47d0d69b8c"
server_key_iv: "92cf2d29ceeca7fdede83b80ae3a2feb"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.472462 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:13.488170 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:13.491024 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:13.491228 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:13.491899 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:13.492733 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:13.492800 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.492871 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:13.492911 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.502250 13218 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:39457
I20260620 19:01:13.502275 13346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:39457 every 8 connection(s)
I20260620 19:01:13.503140 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:13.513566 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 13218
I20260620 19:01:13.513679 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:01:13.513964 2882 external_mini_cluster.cc:1428] Setting key 3ffe702cf4530db487b8206dfafcb1a6
I20260620 19:01:13.516976 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.504969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:38753 (local address 127.2.208.190:37883)
0620 19:01:13.505133 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:13.505138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:13.505969 (+ 831us) server_negotiation.cc:407] Connection header received
0620 19:01:13.507277 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:13.507280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:13.507326 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:13.507427 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:13.508591 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.509138 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.510266 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.510488 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.510565 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.512961 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:13.512964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:13.512966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:13.512989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:13.514774 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.515303 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.515306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.515307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.515428 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.515748 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.515752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.515754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.515994 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:13.516096 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:13.516583 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 19:01:13.516725 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:01:13.517729 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.505377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37883 (local address 127.2.208.129:38753)
0620 19:01:13.505811 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:13.505856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:13.507075 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:13.507589 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:13.507596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:13.507960 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:13.508456 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.508470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.509240 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.509242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.510155 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.510165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.510614 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.510616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.510755 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.511339 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:13.511354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:13.512803 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:13.514895 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.514901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.514916 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.515195 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.515508 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.515511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.515512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.515649 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.516092 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:13.516097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:13.516347 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 19:01:13.516507 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 19:01:13.516767 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260620 19:01:13.519012 13347 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37883
I20260620 19:01:13.519256 13347 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:13.519806 13347 heartbeater.cc:511] Master 127.2.208.190:37883 requested a full tablet report, sending...
I20260620 19:01:13.521296 13156 ts_manager.cc:194] Registered new tserver with Master: bf2aceb5d7c04ca5b4f0d07cc1290aa7 (127.2.208.129:39457)
I20260620 19:01:13.522397 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:38753
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:01:13.541939 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:13.652884 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:13.653132 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:13.653177 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:13.657563 13352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:13.657639 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:13.657727 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:01:13.664211 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:13.665323 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:13.666374 13352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:13.675884 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.675931 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.675889 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:13.676046 13352 server_base.cc:1034] running on GCE node
I20260620 19:01:13.676673 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:13.677295 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:13.678462 13352 hybrid_clock.cc:648] HybridClock initialized: now 1781982073678433 us; error 44 us; skew 500 ppm
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:13.681344 13352 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:01:13.682220 13352 webserver.cc:460] Webserver started at http://127.2.208.130:45177/ using document root <none> and password file <none>
I20260620 19:01:13.682749 13352 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:13.682803 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:13.682963 13352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:13.684978 13352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "9237609967ac44908945afd945faf5e3"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "659cde5ff39b1962d3b25fba2a758a1c"
server_key_iv: "1b97734ad949b85bad9c340757acc08b"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.685492 13352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "9237609967ac44908945afd945faf5e3"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "659cde5ff39b1962d3b25fba2a758a1c"
server_key_iv: "1b97734ad949b85bad9c340757acc08b"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.689150 13352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:01:13.691628 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.692957 13352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:01:13.693181 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "9237609967ac44908945afd945faf5e3"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "659cde5ff39b1962d3b25fba2a758a1c"
server_key_iv: "1b97734ad949b85bad9c340757acc08b"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.693305 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:13.724581 13352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:13.727691 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:13.727905 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:13.728507 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:13.729342 13352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:13.729389 13352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.729429 13352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:13.729485 13352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.738063 13352 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:41907
I20260620 19:01:13.738083 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:41907 every 8 connection(s)
I20260620 19:01:13.738938 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:13.748895 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 13352
I20260620 19:01:13.749018 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:01:13.749291 2882 external_mini_cluster.cc:1428] Setting key 4fb6f475d9b13348f9987590005fa036
I20260620 19:01:13.753008 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.740847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:57411 (local address 127.2.208.190:37883)
0620 19:01:13.741007 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:13.741012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:13.741945 (+ 933us) server_negotiation.cc:407] Connection header received
0620 19:01:13.743563 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:13.743567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:13.743619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:13.743738 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:13.744974 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.745484 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.746599 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.746815 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.746892 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.749377 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:13.749385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:13.749388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:13.749410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:13.750779 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.751257 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.751260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.751262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.751341 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.751753 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.751755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.751756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.751912 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:13.751984 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:13.752657 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 19:01:13.752802 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:01:13.753733 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.741245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37883 (local address 127.2.208.130:57411)
0620 19:01:13.741800 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:13.741843 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:13.743310 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:13.743938 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:13.743948 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:13.744355 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:13.744852 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.744865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.745601 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.745604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.746493 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.746502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.746941 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.746944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.747084 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.747735 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:13.747751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:13.749213 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:13.750885 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.750890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.750904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.751148 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.751459 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.751465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.751469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.751649 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.752009 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:13.752018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:13.752339 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 19:01:13.752565 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 19:01:13.752842 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260620 19:01:13.754959 13481 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37883
I20260620 19:01:13.755187 13481 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:13.755731 13481 heartbeater.cc:511] Master 127.2.208.190:37883 requested a full tablet report, sending...
I20260620 19:01:13.756759 13156 ts_manager.cc:194] Registered new tserver with Master: 9237609967ac44908945afd945faf5e3 (127.2.208.130:41907)
I20260620 19:01:13.757356 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:57411
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:01:13.776209 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:13.884101 13486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:13.884366 13486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:13.884408 13486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:13.888409 13486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:13.888463 13486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:13.888556 13486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:01:13.893771 13486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:13.894788 13486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:13.895828 13486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:13.905349 13491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.905349 13492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:13.905349 13494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:13.905750 13486 server_base.cc:1034] running on GCE node
I20260620 19:01:13.906126 13486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:13.906737 13486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:13.907924 13486 hybrid_clock.cc:648] HybridClock initialized: now 1781982073907912 us; error 30 us; skew 500 ppm
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:13.910718 13486 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:01:13.911625 13486 webserver.cc:460] Webserver started at http://127.2.208.131:44039/ using document root <none> and password file <none>
I20260620 19:01:13.912164 13486 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:13.912248 13486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:13.912470 13486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:01:13.914456 13486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "6096d0d4ad1c47aead8870eb84bf8c3c"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "56a76dcc3a87303f1ec8e8f138c48408"
server_key_iv: "237483a99de7f4d8b8b777b1f7865c9f"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.915005 13486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "6096d0d4ad1c47aead8870eb84bf8c3c"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "56a76dcc3a87303f1ec8e8f138c48408"
server_key_iv: "237483a99de7f4d8b8b777b1f7865c9f"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.918469 13486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:01:13.920791 13501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.922036 13486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:01:13.922160 13486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "6096d0d4ad1c47aead8870eb84bf8c3c"
format_stamp: "Formatted at 2026-06-20 19:01:13 on dist-test-slave-tsm4"
server_key: "56a76dcc3a87303f1ec8e8f138c48408"
server_key_iv: "237483a99de7f4d8b8b777b1f7865c9f"
server_key_version: "encryptionkey@0"
I20260620 19:01:13.922271 13486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:13.941862 13486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:13.945416 13486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:13.945623 13486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:13.946210 13486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:13.947077 13486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:13.947147 13486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.947214 13486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:13.947264 13486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:13.955734 13486 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:46363
I20260620 19:01:13.955775 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:46363 every 8 connection(s)
I20260620 19:01:13.956676 13486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:01:13.962105 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 13486
I20260620 19:01:13.962247 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:01:13.962533 2882 external_mini_cluster.cc:1428] Setting key 7c8d47e610ad1a1534e2c2db12eeae22
Jun 20 19:01:13 dist-test-slave-tsm4 krb5kdc[13111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:13.969934 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.958447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:33975 (local address 127.2.208.190:37883)
0620 19:01:13.958595 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:13.958600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:13.959502 (+ 902us) server_negotiation.cc:407] Connection header received
0620 19:01:13.960770 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:13.960773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:13.960817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:13.960891 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:13.962026 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.962675 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.964023 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.964232 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.964326 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.966681 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:13.966685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:13.966687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:13.966710 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:13.968252 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.968714 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.968717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.968718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.968790 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.969068 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.969071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.969072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.969241 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:13.969313 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:13.969733 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 19:01:13.969822 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:01:13.970774 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.958877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37883 (local address 127.2.208.131:33975)
0620 19:01:13.959305 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:13.959346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:13.960585 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:13.961044 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:13.961050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:13.961388 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:13.961856 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.961869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.962807 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.962810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.963919 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:13.963930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.964387 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.964391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:13.964578 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.965150 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:13.965166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:13.966533 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:13.968347 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.968368 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.968379 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.968612 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.968867 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:13.968869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:13.968870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:13.968993 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:13.969323 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:13.969328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:13.969537 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 19:01:13.969687 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 19:01:13.969922 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260620 19:01:13.971897 13615 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37883
I20260620 19:01:13.972122 13615 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:13.972579 13615 heartbeater.cc:511] Master 127.2.208.190:37883 requested a full tablet report, sending...
I20260620 19:01:13.973583 13156 ts_manager.cc:194] Registered new tserver with Master: 6096d0d4ad1c47aead8870eb84bf8c3c (127.2.208.131:46363)
I20260620 19:01:13.974099 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:33975
I20260620 19:01:13.976419 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:01:13.985728 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.978253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.2.208.190:37883)
0620 19:01:13.978377 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:13.978380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:13.978596 (+ 216us) server_negotiation.cc:407] Connection header received
0620 19:01:13.978687 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:13.978689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:13.978724 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:13.978799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:13.979497 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.980012 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.980857 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:13.981087 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:13.981191 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:13.982511 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:13.982515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:13.982518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:13.982542 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:13.984075 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.984507 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.984512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.984514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.984596 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:13.984874 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:13.984878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:13.984879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:13.985085 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:13.985229 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:13.985471 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:01:13.985579 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:01:13.988454 13156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:01:13.991590 13156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:01:14.006247 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.000556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.2.208.130:41907)
0620 19:01:14.000915 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.000929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.001050 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:01:14.001147 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.001157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.001383 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.001582 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:14.001957 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.003284 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.005045 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.005595 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.005725 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.005879 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.005991 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.006077 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":149,"threads_started":1}
I20260620 19:01:14.006247 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.000410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:41907 (local address 127.0.0.1:33048)
0620 19:01:14.000942 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.000962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:14.001030 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:14.001681 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:14.001688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:14.001715 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:14.001837 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.001842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.003455 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.003461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.004915 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.004927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.005723 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.005727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.005826 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.005859 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:01:14.005965 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:01:14.006077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":95,"threads_started":1}
I20260620 19:01:14.006811 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.999737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39457 (local address 127.0.0.1:39356)
0620 19:01:14.000385 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.000438 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:14.000574 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:14.002419 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:14.002421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:14.002435 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:14.002505 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.002509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.004007 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.004011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.005126 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.005136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.006411 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.006414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.006510 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.006535 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:01:14.006586 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:01:14.006671 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":74,"threads_started":1}
I20260620 19:01:14.007512 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.000371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.2.208.129:39457)
0620 19:01:14.001938 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.001954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.001975 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:01:14.002050 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.002055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.002257 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.002442 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:14.002583 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.003833 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.005375 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.006257 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.007005 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.007155 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.007212 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.007326 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":100,"threads_started":1}
I20260620 19:01:14.008704 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:13.999969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:46363 (local address 127.0.0.1:40094)
0620 19:01:14.000709 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.000725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:14.000816 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:14.003539 (+ 2723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:14.003542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:14.003563 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:14.003650 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.003656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.005001 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.005005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.006273 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.006285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.008312 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.008316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.008406 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.008428 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:01:14.008512 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:01:14.008574 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":155,"threads_started":1}
I20260620 19:01:14.008795 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.000750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.2.208.131:46363)
0620 19:01:14.001299 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.001311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.001331 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:01:14.001402 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.001407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.001611 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.003579 (+ 1968us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:14.003730 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.004858 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.006416 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.008180 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.008310 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.008448 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.008535 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.008647 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":97,"threads_started":1}
I20260620 19:01:14.009801 13415 tablet_service.cc:1467] Processing CreateTablet for tablet c675e4fef6054dbea6f1ef29da1228fe (DEFAULT_TABLE table=test-table [id=017c532f30254f8698b98b8e9bc12668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:14.010278 13281 tablet_service.cc:1467] Processing CreateTablet for tablet c675e4fef6054dbea6f1ef29da1228fe (DEFAULT_TABLE table=test-table [id=017c532f30254f8698b98b8e9bc12668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:14.010793 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c675e4fef6054dbea6f1ef29da1228fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:14.011541 13281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c675e4fef6054dbea6f1ef29da1228fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:14.011746 13549 tablet_service.cc:1467] Processing CreateTablet for tablet c675e4fef6054dbea6f1ef29da1228fe (DEFAULT_TABLE table=test-table [id=017c532f30254f8698b98b8e9bc12668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:14.012745 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c675e4fef6054dbea6f1ef29da1228fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:14.016897 13632 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Bootstrap starting.
I20260620 19:01:14.018950 13633 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: Bootstrap starting.
I20260620 19:01:14.019619 13632 tablet_bootstrap.cc:654] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:14.020579 13632 log.cc:824] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:14.021081 13634 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Bootstrap starting.
I20260620 19:01:14.021147 13633 tablet_bootstrap.cc:654] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:14.021936 13633 log.cc:824] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:14.022610 13632 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: No bootstrap required, opened a new log
I20260620 19:01:14.022821 13632 ts_tablet_manager.cc:1397] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:01:14.023485 13634 tablet_bootstrap.cc:654] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:14.023929 13633 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: No bootstrap required, opened a new log
I20260620 19:01:14.024104 13633 ts_tablet_manager.cc:1397] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:01:14.024281 13634 log.cc:824] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:14.026281 13634 tablet_bootstrap.cc:492] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: No bootstrap required, opened a new log
I20260620 19:01:14.026468 13634 ts_tablet_manager.cc:1397] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:01:14.026921 13632 raft_consensus.cc:348] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.027186 13632 raft_consensus.cc:374] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:14.027256 13632 raft_consensus.cc:729] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9237609967ac44908945afd945faf5e3, State: Initialized, Role: FOLLOWER
I20260620 19:01:14.027719 13632 consensus_queue.cc:260] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.028251 13633 raft_consensus.cc:348] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.028422 13481 heartbeater.cc:503] Master 127.2.208.190:37883 was elected leader, sending a full tablet report...
I20260620 19:01:14.028489 13633 raft_consensus.cc:374] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:14.028537 13633 raft_consensus.cc:729] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2aceb5d7c04ca5b4f0d07cc1290aa7, State: Initialized, Role: FOLLOWER
I20260620 19:01:14.028753 13632 ts_tablet_manager.cc:1428] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:01:14.028996 13633 consensus_queue.cc:260] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.029639 13347 heartbeater.cc:503] Master 127.2.208.190:37883 was elected leader, sending a full tablet report...
I20260620 19:01:14.029913 13633 ts_tablet_manager.cc:1428] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:01:14.031564 13634 raft_consensus.cc:348] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.031828 13634 raft_consensus.cc:374] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:14.031899 13634 raft_consensus.cc:729] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6096d0d4ad1c47aead8870eb84bf8c3c, State: Initialized, Role: FOLLOWER
I20260620 19:01:14.032337 13634 consensus_queue.cc:260] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.032963 13615 heartbeater.cc:503] Master 127.2.208.190:37883 was elected leader, sending a full tablet report...
I20260620 19:01:14.033299 13634 ts_tablet_manager.cc:1428] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:01:14.103662 13640 raft_consensus.cc:482] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:01:14.103873 13640 raft_consensus.cc:504] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.105113 13640 leader_election.cc:288] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2aceb5d7c04ca5b4f0d07cc1290aa7 (127.2.208.129:39457), 9237609967ac44908945afd945faf5e3 (127.2.208.130:41907)
I20260620 19:01:14.108657 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.105509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:33527 (local address 127.2.208.129:39457)
0620 19:01:14.105645 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.105650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.105665 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:01:14.105722 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.105725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.105778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.105874 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:14.106156 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.106678 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.107710 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.108107 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.108237 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.108291 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.108462 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.108527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:01:14.108662 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.105376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39457 (local address 127.2.208.131:33527)
0620 19:01:14.105504 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.105520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:14.105615 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:14.105896 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:14.105899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:14.105918 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:14.106037 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.106043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.106785 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.106791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.107581 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.107589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.108223 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.108226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.108332 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.108359 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:01:14.108447 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:01:14.108516 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:01:14.109217 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.105808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:40605 (local address 127.2.208.130:41907)
0620 19:01:14.105939 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.105945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.106162 (+ 217us) server_negotiation.cc:407] Connection header received
0620 19:01:14.106311 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.106314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.106355 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.106437 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:14.106721 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.107188 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.108240 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.108641 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.108770 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.108815 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.109038 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.109107 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:01:14.109309 13301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c675e4fef6054dbea6f1ef29da1228fe" candidate_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" is_pre_election: true
I20260620 19:01:14.109560 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.105659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:41907 (local address 127.2.208.131:40605)
0620 19:01:14.106071 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.106081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:14.106191 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:14.106443 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:14.106446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:14.106460 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:14.106613 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.106618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.107296 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.107302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.108108 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:14.108116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.108777 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.108780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:14.108887 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.108919 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 19:01:14.109363 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 19:01:14.109439 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":139,"threads_started":1}
I20260620 19:01:14.109648 13301 raft_consensus.cc:2454] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6096d0d4ad1c47aead8870eb84bf8c3c in term 0.
I20260620 19:01:14.110086 13503 leader_election.cc:302] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6096d0d4ad1c47aead8870eb84bf8c3c, bf2aceb5d7c04ca5b4f0d07cc1290aa7; no voters:
I20260620 19:01:14.110131 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c675e4fef6054dbea6f1ef29da1228fe" candidate_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9237609967ac44908945afd945faf5e3" is_pre_election: true
I20260620 19:01:14.110386 13640 raft_consensus.cc:2781] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:01:14.110420 13435 raft_consensus.cc:2454] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6096d0d4ad1c47aead8870eb84bf8c3c in term 0.
I20260620 19:01:14.110481 13640 raft_consensus.cc:482] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:01:14.110553 13640 raft_consensus.cc:3037] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:14.111645 13640 raft_consensus.cc:504] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.111990 13640 leader_election.cc:288] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [CANDIDATE]: Term 1 election: Requested vote from peers bf2aceb5d7c04ca5b4f0d07cc1290aa7 (127.2.208.129:39457), 9237609967ac44908945afd945faf5e3 (127.2.208.130:41907)
I20260620 19:01:14.112370 13301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c675e4fef6054dbea6f1ef29da1228fe" candidate_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7"
I20260620 19:01:14.112466 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c675e4fef6054dbea6f1ef29da1228fe" candidate_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9237609967ac44908945afd945faf5e3"
I20260620 19:01:14.112506 13301 raft_consensus.cc:3037] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:14.112619 13435 raft_consensus.cc:3037] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:14.113566 13301 raft_consensus.cc:2454] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6096d0d4ad1c47aead8870eb84bf8c3c in term 1.
I20260620 19:01:14.113716 13435 raft_consensus.cc:2454] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6096d0d4ad1c47aead8870eb84bf8c3c in term 1.
I20260620 19:01:14.113847 13503 leader_election.cc:302] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6096d0d4ad1c47aead8870eb84bf8c3c, bf2aceb5d7c04ca5b4f0d07cc1290aa7; no voters:
I20260620 19:01:14.114032 13640 raft_consensus.cc:2781] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:14.114301 13640 raft_consensus.cc:686] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 1 LEADER]: Becoming Leader. State: Replica: 6096d0d4ad1c47aead8870eb84bf8c3c, State: Running, Role: LEADER
I20260620 19:01:14.114624 13640 consensus_queue.cc:237] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } }
I20260620 19:01:14.117892 13155 catalog_manager.cc:5495] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6096d0d4ad1c47aead8870eb84bf8c3c (127.2.208.131). New cstate: current_term: 1 leader_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6096d0d4ad1c47aead8870eb84bf8c3c" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 } health_report { overall_health: UNKNOWN } } }
I20260620 19:01:14.131215 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.127589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.2.208.131:46363)
0620 19:01:14.127710 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:14.127715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:14.127759 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:01:14.127948 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:14.127951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:14.128013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:14.128122 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:01:14.128380 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.128907 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.129773 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:14.130005 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:14.130114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:14.130434 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:01:14.130833 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:01:14.130963 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:14.131020 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:01:14.131074 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:01:14.136715 13435 raft_consensus.cc:1261] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 1 FOLLOWER]: Refusing update from remote peer 6096d0d4ad1c47aead8870eb84bf8c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:01:14.136739 13301 raft_consensus.cc:1261] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 1 FOLLOWER]: Refusing update from remote peer 6096d0d4ad1c47aead8870eb84bf8c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:01:14.137408 13640 consensus_queue.cc:1035] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9237609967ac44908945afd945faf5e3" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:01:14.137621 13642 consensus_queue.cc:1035] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2aceb5d7c04ca5b4f0d07cc1290aa7" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:01:14.147217 13646 mvcc.cc:205] Tried to move back new op lower bound from 7298998575657222144 to 7298998575576104960. Current Snapshot: MvccSnapshot[applied={T|T < 7298998575657222144}]
I20260620 19:01:14.147651 13647 mvcc.cc:205] Tried to move back new op lower bound from 7298998575657222144 to 7298998575576104960. Current Snapshot: MvccSnapshot[applied={T|T < 7298998575657222144 or (T in {7298998575657222144})}]
I20260620 19:01:14.152669 13156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:01:14.152813 13156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:01:14.155196 13156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a030263fe1974e10bbd6cf5699a9f6b9: Sending DeleteTablet for 3 replicas of tablet c675e4fef6054dbea6f1ef29da1228fe
I20260620 19:01:14.155970 13549 tablet_service.cc:1514] Processing DeleteTablet for tablet c675e4fef6054dbea6f1ef29da1228fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:14 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:40094
I20260620 19:01:14.156211 13281 tablet_service.cc:1514] Processing DeleteTablet for tablet c675e4fef6054dbea6f1ef29da1228fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:14 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39356
I20260620 19:01:14.156347 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet c675e4fef6054dbea6f1ef29da1228fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:14 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:33048
I20260620 19:01:14.156440 13654 tablet_replica.cc:330] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: stopping tablet replica
I20260620 19:01:14.156639 13655 tablet_replica.cc:330] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7: stopping tablet replica
I20260620 19:01:14.156744 13654 raft_consensus.cc:2229] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:01:14.157011 13655 raft_consensus.cc:2229] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:01:14.157073 13654 raft_consensus.cc:2258] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:01:14.157284 13655 raft_consensus.cc:2258] T c675e4fef6054dbea6f1ef29da1228fe P bf2aceb5d7c04ca5b4f0d07cc1290aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:01:14.157598 13656 tablet_replica.cc:330] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: stopping tablet replica
I20260620 19:01:14.157814 13656 raft_consensus.cc:2229] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:01:14.158041 13656 raft_consensus.cc:2258] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:01:14.158532 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 13218
I20260620 19:01:14.160548 13656 ts_tablet_manager.cc:1905] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:01:14.160548 13654 ts_tablet_manager.cc:1905] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:01:14.163328 13654 ts_tablet_manager.cc:1918] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:01:14.163463 13654 log.cc:1196] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/c675e4fef6054dbea6f1ef29da1228fe
I20260620 19:01:14.163539 13656 ts_tablet_manager.cc:1918] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:01:14.163621 13656 log.cc:1196] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/c675e4fef6054dbea6f1ef29da1228fe
I20260620 19:01:14.163807 13654 ts_tablet_manager.cc:1939] T c675e4fef6054dbea6f1ef29da1228fe P 6096d0d4ad1c47aead8870eb84bf8c3c: Deleting consensus metadata
I20260620 19:01:14.164085 13656 ts_tablet_manager.cc:1939] T c675e4fef6054dbea6f1ef29da1228fe P 9237609967ac44908945afd945faf5e3: Deleting consensus metadata
I20260620 19:01:14.165037 13140 catalog_manager.cc:4841] TS 6096d0d4ad1c47aead8870eb84bf8c3c (127.2.208.131:46363): tablet c675e4fef6054dbea6f1ef29da1228fe (table test-table [id=017c532f30254f8698b98b8e9bc12668]) successfully deleted
I20260620 19:01:14.165114 13141 catalog_manager.cc:4841] TS 9237609967ac44908945afd945faf5e3 (127.2.208.130:41907): tablet c675e4fef6054dbea6f1ef29da1228fe (table test-table [id=017c532f30254f8698b98b8e9bc12668]) successfully deleted
W20260620 19:01:14.167204 13141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:39457 (error 108)
I20260620 19:01:14.167445 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 13352
I20260620 19:01:14.168709 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:14.168419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39457 (local address 127.0.0.1:39366)
0620 19:01:14.168549 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:14.168631 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 19:01:14.168936 13141 catalog_manager.cc:4584] TS bf2aceb5d7c04ca5b4f0d07cc1290aa7 (127.2.208.129:39457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c675e4fef6054dbea6f1ef29da1228fe: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39457: connect: Connection refused (error 111)
I20260620 19:01:14.175534 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 13486
I20260620 19:01:14.181557 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 13124
2026-06-20T19:01:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2292 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:01:14 dist-test-slave-tsm4 krb5kdc[13660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:01:14 dist-test-slave-tsm4 krb5kdc[13660](info): set up 2 sockets
Jun 20 19:01:14 dist-test-slave-tsm4 krb5kdc[13660](info): commencing operation
krb5kdc: starting...
W20260620 19:01:15.232297 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:01:15 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:01:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangeradmin_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangeradmin_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangerlookup_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangerlookup_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
I20260620 19:01:15.396224 2882 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDDC2sW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/postgres -l logfile start
2026-06-20 19:01:17.710 UTC [13690] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:01:17.710 UTC [13690] LOG: listening on IPv4 address "127.2.208.148", port 58599
2026-06-20 19:01:17.713 UTC [13690] LOG: listening on Unix socket "/tmp/.s.PGSQL.58599"
2026-06-20 19:01:17.733 UTC [13693] LOG: database system was shut down at 2026-06-20 19:01:16 UTC
2026-06-20 19:01:17.737 UTC [13690] LOG: database system is ready to accept connections
I20260620 19:01:18.696056 2882 mini_postgres.cc:96] Postgres bound to 58599
2026-06-20 19:01:18.702 UTC [13701] FATAL: database "slave" does not exist
127.2.208.148:58599 - accepting connections
I20260620 19:01:18.702855 2882 mini_ranger.cc:161] Starting Ranger...
I20260620 19:01:18.717651 2882 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:01:19.147831 2882 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:01:19.147927 2882 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin
2026-06-20 19:01:19,425 [I] DB FLAVOR :POSTGRES
2026-06-20 19:01:19,426 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:01:19,426 [I] Checking connection..
2026-06-20 19:01:19,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:01:19,676 [I] Checking connection passed.
2026-06-20 19:01:19,676 [I] --------- Verifying version history table ---------
2026-06-20 19:01:19,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:01:19,941 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:01:19,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:01:20,191 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:01:20,191 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:01:20,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:01:20.418 UTC [13765] WARNING: there is no transaction in progress
2026-06-20 19:01:20,424 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:01:20,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:01:20,691 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:01:20,691 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:01:20,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:01:20,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:01:21,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tsm4.c.gcp-upstream.internal','N') ;"
2026-06-20 19:01:21,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:01:21,600 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:01:21,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:01:21,827 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:01:21,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:01:22,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:01:22,324 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:01:22,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:01:23.243 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.259 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.275 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.289 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.304 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.324 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.382 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.389 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.403 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.412 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.422 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.435 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.444 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.450 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.455 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:23.461 UTC [13873] WARNING: there is no transaction in progress
2026-06-20 19:01:24,031 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:01:24,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:01:24,246 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:01:24,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:01:24,491 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:01:24,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:01:24,727 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:01:24,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:01:24,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:01:25,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tsm4.c.gcp-upstream.internal';"
2026-06-20 19:01:25,422 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:01:25,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:01:25,670 [I] DB_PATCHES have already been applied
I20260620 19:01:25.677040 2882 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDDC2sW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:01:25.677104 2882 mini_ranger.cc:193] Using host: 127.2.208.148
I20260620 19:01:25.680624 2882 mini_ranger.cc:227] Ranger admin URL: http://127.2.208.148:41327
Jun 20, 2026 7:01:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:01:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:01:26.780084 2882 mini_ranger.cc:160] Time spent starting Ranger: real 8.077s user 0.003s sys 0.007s
Jun 20, 2026 7:01:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:01:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 7:01:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:01:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:01:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:01:46.461792 2882 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangerkms_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/rangerkms_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.208.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.208.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:01:46 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:01:46.539433 2882 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:01:46.548178 2882 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:01:47.326099 2882 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:01:47.326193 2882 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms
I20260620 19:01:47.326284 2882 mini_ranger_kms.cc:227] Using postgres at 127.2.208.148:58599
2026-06-20 19:01:47,512 [I] DB FLAVOR :POSTGRES
2026-06-20 19:01:47,512 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:01:47,512 [I] Checking connection
2026-06-20 19:01:47,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:01:47,764 [I] connection success
2026-06-20 19:01:47,764 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:01:47,764 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:01:47,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:01:48,012 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:01:48,012 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:01:48,012 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:01:48,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:58599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:01:48,253 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:01:48.693970 2882 mini_ranger_kms.cc:310] Created kms service
I20260620 19:01:48.879989 2882 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:01:48.918598 2882 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:01:48 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:49.301540 2882 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:01:49.301694 2882 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:01:49.301718 2882 mini_ranger_kms.cc:240] Using host: 127.2.208.148
I20260620 19:01:49.305689 2882 mini_ranger_kms.cc:276] Ranger KMS PID: 14053
I20260620 19:01:49.305836 2882 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.208.148:46217
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 19:01:51 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:51.540120 2882 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.001s user 0.018s sys 0.152s
I20260620 19:01:51.540264 2882 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:01:51.540385 2882 mini_ranger_kms.cc:401] 127.2.208.148:46217/kms/v1/keys
Jun 20 19:01:51 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
Jun 20 19:01:52 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982111, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
Jun 20 19:01:54 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:01:54.216073 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37979
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:37979
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:46217/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:54.324606 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:54.324884 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:54.324950 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:54.328922 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:01:54.328996 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:54.329034 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:54.329084 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:01:54.329119 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:01:54.334367 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.208.148:46217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:37979
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:54.335366 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:54.336344 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:01:54.345310 14104 server_base.cc:1034] running on GCE node
W20260620 19:01:54.345268 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:54.345278 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:54.345321 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:54.346098 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:54.347078 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:54.348265 14104 hybrid_clock.cc:648] HybridClock initialized: now 1781982114348228 us; error 49 us; skew 500 ppm
Jun 20 19:01:54 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982114, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:54.355796 14104 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:01:54.356654 14104 webserver.cc:460] Webserver started at http://127.2.208.190:42665/ using document root <none> and password file <none>
I20260620 19:01:54.357149 14104 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:54.357223 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:54.357434 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:01:54 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982114, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:54.402369 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "d274b34f763641018d241d09eee5555b"
format_stamp: "Formatted at 2026-06-20 19:01:54 on dist-test-slave-tsm4"
server_key: "17d188d6f552001c9302ab95e780263f"
server_key_iv: "259f675eb7c624ddd9c2c1e420241377"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:54.403023 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "d274b34f763641018d241d09eee5555b"
format_stamp: "Formatted at 2026-06-20 19:01:54 on dist-test-slave-tsm4"
server_key: "17d188d6f552001c9302ab95e780263f"
server_key_iv: "259f675eb7c624ddd9c2c1e420241377"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:54.406912 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:01:54.420487 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:54.421985 14104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:01:54.422124 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "d274b34f763641018d241d09eee5555b"
format_stamp: "Formatted at 2026-06-20 19:01:54 on dist-test-slave-tsm4"
server_key: "17d188d6f552001c9302ab95e780263f"
server_key_iv: "259f675eb7c624ddd9c2c1e420241377"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:54.422256 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:54.435485 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:54.438102 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:54.438293 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:54.446568 14104 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:37979
I20260620 19:01:54.446614 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:37979 every 8 connection(s)
I20260620 19:01:54.447579 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:01:54.450811 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:54.452494 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 14104
I20260620 19:01:54.452613 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:01:54.452879 2882 external_mini_cluster.cc:1428] Setting key 3dfba2fcdf782a36b92881bfcdaa0c15
I20260620 19:01:54.457659 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: Bootstrap starting.
I20260620 19:01:54.460198 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:54.461066 14172 log.cc:824] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:01:54 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:54.463076 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: No bootstrap required, opened a new log
I20260620 19:01:54.466795 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } }
I20260620 19:01:54.467046 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:54.467136 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d274b34f763641018d241d09eee5555b, State: Initialized, Role: FOLLOWER
I20260620 19:01:54.467636 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } }
I20260620 19:01:54.467792 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:01:54.467860 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:54.454222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.2.208.190:37979)
0620 19:01:54.454576 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:54.454597 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:54.454630 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:01:54.455279 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:54.455295 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:54.455696 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:54.456004 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:54.456777 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:54.457758 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:54.458613 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:54.459056 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:54.459214 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:54.461715 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:54.461721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:54.461737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:54.461764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:54.464706 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:54.465198 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:54.465204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:54.465210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:54.465347 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:54.465655 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:54.465658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:54.465660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:54.466143 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:54.466334 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:54.466603 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:01:54.466874 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
I20260620 19:01:54.467869 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:01:54.468042 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:54.468906 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } }
I20260620 19:01:54.469276 14172 leader_election.cc:302] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d274b34f763641018d241d09eee5555b; no voters:
I20260620 19:01:54.469631 14172 leader_election.cc:288] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:01:54.469736 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:54.469980 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [term 1 LEADER]: Becoming Leader. State: Replica: d274b34f763641018d241d09eee5555b, State: Running, Role: LEADER
I20260620 19:01:54.470284 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } }
I20260620 19:01:54.470847 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:01:54.471416 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d274b34f763641018d241d09eee5555b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } } }
I20260620 19:01:54.471593 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:54.471874 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d274b34f763641018d241d09eee5555b. Latest consensus state: current_term: 1 leader_uuid: "d274b34f763641018d241d09eee5555b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d274b34f763641018d241d09eee5555b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 37979 } } }
I20260620 19:01:54.471997 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b [sys.catalog]: This master's current role is: LEADER
I20260620 19:01:54.472291 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:01:54.475751 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:01:54.476153 14172 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:01:54.486650 14186 catalog_manager.cc:1347] Generated new cluster ID: 1931ab1ffda6456588444b5f5633a875
I20260620 19:01:54.486793 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:01:54.507598 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:01:54.508517 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:01:54.529237 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: Generated new TSK 0
I20260620 19:01:54.530081 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:01:55.182351 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:55.297624 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:55.297894 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:55.297982 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:55.301839 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:55.301936 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:55.302018 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:01:55.307157 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.208.148:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:55.308187 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:55.309199 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:55.319839 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.320047 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.320574 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:55.320930 14217 server_base.cc:1034] running on GCE node
I20260620 19:01:55.321768 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:55.322526 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:55.325783 14217 hybrid_clock.cc:648] HybridClock initialized: now 1781982115325758 us; error 34 us; skew 500 ppm
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:55.329241 14217 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:01:55.330319 14217 webserver.cc:460] Webserver started at http://127.2.208.129:39659/ using document root <none> and password file <none>
I20260620 19:01:55.331403 14217 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:55.331486 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:55.331710 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:55.346148 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "791bd91319584a25b1b8ea13ba86fd4f"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "98f2b83a65ed4b6e5863c4ced080801e"
server_key_iv: "2e94236480dd3c9b39a5739d91ed2d81"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.346881 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "791bd91319584a25b1b8ea13ba86fd4f"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "98f2b83a65ed4b6e5863c4ced080801e"
server_key_iv: "2e94236480dd3c9b39a5739d91ed2d81"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.354835 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260620 19:01:55.370108 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.371861 14217 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:01:55.372002 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "791bd91319584a25b1b8ea13ba86fd4f"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "98f2b83a65ed4b6e5863c4ced080801e"
server_key_iv: "2e94236480dd3c9b39a5739d91ed2d81"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.372143 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:55.409286 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:55.412431 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:55.412621 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:55.413177 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:55.414005 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:55.414067 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.414135 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:55.414173 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.424938 14217 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:34433
I20260620 19:01:55.425068 14347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:34433 every 8 connection(s)
I20260620 19:01:55.426070 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:01:55.430737 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 14217
I20260620 19:01:55.430850 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:01:55.431166 2882 external_mini_cluster.cc:1428] Setting key b2d892104fc761447249eee4faaaaa34
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:55.443534 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:55.428754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37979 (local address 127.2.208.129:58195)
0620 19:01:55.429307 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:55.429362 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:55.430374 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:55.431175 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:55.431184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:55.431568 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:55.432206 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:55.432225 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.433398 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.433401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:55.434782 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:55.434793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.435551 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.435555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:55.435771 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:55.436431 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:55.436452 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:55.438142 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:55.440397 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:55.440403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:55.440417 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:55.440752 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:55.441057 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:55.441061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:55.441063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:55.441228 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:55.441702 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:55.441708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:55.441989 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 19:01:55.442183 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 19:01:55.442490 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260620 19:01:55.444254 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:55.429761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:58195 (local address 127.2.208.190:37979)
0620 19:01:55.430053 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:55.430062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:55.430080 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:01:55.430715 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:55.430719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:55.430776 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:55.430869 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:55.432375 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.433244 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.434919 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.435276 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.435389 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:55.438329 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:55.438332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:55.438335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:55.438360 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:55.440271 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:55.440867 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:55.440871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:55.440873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:55.440959 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:55.441336 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:55.441339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:55.441341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:55.441548 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:55.441640 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:55.443961 (+ 2321us) server_negotiation.cc:299] Negotiation successful
0620 19:01:55.444086 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260620 19:01:55.445896 14348 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37979
I20260620 19:01:55.446122 14348 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:55.446655 14348 heartbeater.cc:511] Master 127.2.208.190:37979 requested a full tablet report, sending...
I20260620 19:01:55.448423 14136 ts_manager.cc:194] Registered new tserver with Master: 791bd91319584a25b1b8ea13ba86fd4f (127.2.208.129:34433)
I20260620 19:01:55.449904 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:58195
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:01:55.482384 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:01:55.607287 14356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:55.607602 14356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:55.607697 14356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:55.611713 14356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:55.611776 14356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:55.611876 14356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:01:55.617969 14356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.208.148:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:55.619139 14356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:55.620326 14356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:55.630288 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.630379 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.630447 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:55.631479 14356 server_base.cc:1034] running on GCE node
I20260620 19:01:55.631875 14356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:55.632542 14356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:55.633710 14356 hybrid_clock.cc:648] HybridClock initialized: now 1781982115633681 us; error 43 us; skew 500 ppm
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:55.636606 14356 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:01:55.637524 14356 webserver.cc:460] Webserver started at http://127.2.208.130:33715/ using document root <none> and password file <none>
I20260620 19:01:55.638056 14356 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:55.638139 14356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:55.638341 14356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:55.650996 14356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "d9d9f60d0f1647e7b401838e8f010609"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "1297d725fe6212efa07f510fb97d163b"
server_key_iv: "53581c893630dfcadf3b9cf192e00a64"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.651731 14356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d9f60d0f1647e7b401838e8f010609"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "1297d725fe6212efa07f510fb97d163b"
server_key_iv: "53581c893630dfcadf3b9cf192e00a64"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.657543 14356 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 19:01:55.676254 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.677910 14356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 19:01:55.678045 14356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "d9d9f60d0f1647e7b401838e8f010609"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "1297d725fe6212efa07f510fb97d163b"
server_key_iv: "53581c893630dfcadf3b9cf192e00a64"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.678205 14356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:55.719189 14356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:55.722776 14356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:55.722959 14356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:55.723632 14356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:55.724617 14356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:55.724676 14356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.724732 14356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:55.724762 14356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:55.735652 14356 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44943
I20260620 19:01:55.736879 14356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:01:55.739589 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44943 every 8 connection(s)
I20260620 19:01:55.742411 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 14356
I20260620 19:01:55.742529 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:01:55.742812 2882 external_mini_cluster.cc:1428] Setting key 38bdfd0fd44838c58a557b2593573c11
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:55.761060 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:55.743058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:55691 (local address 127.2.208.190:37979)
0620 19:01:55.743187 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:55.743192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:55.748197 (+ 5005us) server_negotiation.cc:407] Connection header received
0620 19:01:55.749473 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:55.749477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:55.749530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:55.749581 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:55.751160 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.751930 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.753585 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.753921 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.753995 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:55.756992 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:55.756996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:55.756999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:55.757024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:55.758705 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:55.759311 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:55.759315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:55.759317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:55.759431 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:55.759816 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:55.759820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:55.759821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:55.760023 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:55.760091 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:55.760760 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 19:01:55.760888 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:01:55.762863 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:55.741999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37979 (local address 127.2.208.130:55691)
0620 19:01:55.748008 (+ 6009us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:55.748086 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:55.749224 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:55.749987 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:55.749996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:55.750357 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:55.750999 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:55.751016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.752087 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.752091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:55.753450 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:55.753461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:55.754206 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:55.754210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:55.754398 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:55.755081 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:55.755101 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:55.756810 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:55.758843 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:55.758849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:55.758866 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:55.759195 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:55.759542 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:55.759545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:55.759547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:55.759710 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:55.760169 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:55.760187 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:55.760489 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 19:01:55.761528 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0620 19:01:55.761853 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5736,"thread_start_us":136,"threads_started":1}
I20260620 19:01:55.764389 14486 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37979
I20260620 19:01:55.764631 14486 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:55.765167 14486 heartbeater.cc:511] Master 127.2.208.190:37979 requested a full tablet report, sending...
I20260620 19:01:55.766433 14136 ts_manager.cc:194] Registered new tserver with Master: d9d9f60d0f1647e7b401838e8f010609 (127.2.208.130:44943)
I20260620 19:01:55.767256 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:55691
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:01:55.795537 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:01:55.929796 14491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:01:55.930056 14491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:01:55.930153 14491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:01:55.934791 14491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:01:55.934875 14491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:01:55.934973 14491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:01:55.940178 14491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.208.148:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:01:55.941208 14491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:01:55.942308 14491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:01:55.953555 14497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.954005 14496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:01:55.954006 14499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:01:55.955264 14491 server_base.cc:1034] running on GCE node
I20260620 19:01:55.955783 14491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:01:55.956612 14491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:01:55.957868 14491 hybrid_clock.cc:648] HybridClock initialized: now 1781982115957849 us; error 35 us; skew 500 ppm
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:01:55.961615 14491 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:01:55.962615 14491 webserver.cc:460] Webserver started at http://127.2.208.131:41561/ using document root <none> and password file <none>
I20260620 19:01:55.963312 14491 fs_manager.cc:359] Metadata directory not provided
I20260620 19:01:55.963470 14491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:01:55.963730 14491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:01:55 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:55.981092 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "583080f4b4f9404db5971c39515c9775"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "7c541185b4c91f53eaba5cb37f6acf06"
server_key_iv: "a0f7bcbafb618dacf64572ac950801fb"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.982059 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "583080f4b4f9404db5971c39515c9775"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "7c541185b4c91f53eaba5cb37f6acf06"
server_key_iv: "a0f7bcbafb618dacf64572ac950801fb"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:55.987037 14491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:01:56.003727 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:56.005371 14491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:01:56.005487 14491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "583080f4b4f9404db5971c39515c9775"
format_stamp: "Formatted at 2026-06-20 19:01:55 on dist-test-slave-tsm4"
server_key: "7c541185b4c91f53eaba5cb37f6acf06"
server_key_iv: "a0f7bcbafb618dacf64572ac950801fb"
server_key_version: "kuduclusterkey@0"
I20260620 19:01:56.005600 14491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:01:56.025429 14491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:01:56.029055 14491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:01:56.029286 14491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:01:56.029944 14491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:01:56.031097 14491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:01:56.031204 14491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:56.031294 14491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:01:56.031354 14491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:01:56.042696 14491 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:44669
I20260620 19:01:56.043807 14491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:01:56.044768 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 14491
I20260620 19:01:56.044890 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:01:56.045199 2882 external_mini_cluster.cc:1428] Setting key 567e3baf9ee33579c09076995540e52c
Jun 20 19:01:56 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:01:56.045939 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:44669 every 8 connection(s)
Jun 20 19:01:56 dist-test-slave-tsm4 krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:01:56.067899 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.049127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:52793 (local address 127.2.208.190:37979)
0620 19:01:56.049252 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.049256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.051867 (+ 2611us) server_negotiation.cc:407] Connection header received
0620 19:01:56.053022 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.053025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.053076 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.053135 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:56.054715 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.055653 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.059897 (+ 4244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.060241 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.060309 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.063198 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:56.063202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:56.063205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:56.063230 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:56.064941 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:56.065900 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:56.065915 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:56.065917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:56.066098 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:56.066624 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:56.066627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:56.066629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:56.066832 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:56.066916 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.067605 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.067737 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:01:56.071043 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.048137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:37979 (local address 127.2.208.131:52793)
0620 19:01:56.051725 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.051777 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.052793 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.053526 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.053535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.053911 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:01:56.054561 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.054578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.055783 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.055788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.059769 (+ 3981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.059779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.060491 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.060494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.060674 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.061307 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:01:56.061326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:01:56.063008 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:01:56.065067 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:56.065073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:56.065089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:56.065666 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:56.066327 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:01:56.066330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:01:56.066332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:01:56.066522 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:01:56.066998 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:01:56.067005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:01:56.067305 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.068293 (+ 988us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.068612 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3313,"thread_start_us":129,"threads_started":1}
I20260620 19:01:56.072614 14620 heartbeater.cc:347] Connected to a master server at 127.2.208.190:37979
I20260620 19:01:56.072849 14620 heartbeater.cc:464] Registering TS with master...
I20260620 19:01:56.073403 14620 heartbeater.cc:511] Master 127.2.208.190:37979 requested a full tablet report, sending...
I20260620 19:01:56.074455 14136 ts_manager.cc:194] Registered new tserver with Master: 583080f4b4f9404db5971c39515c9775 (127.2.208.131:44669)
I20260620 19:01:56.075143 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:52793
I20260620 19:01:56.081593 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:01:56.092417 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.083678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.2.208.190:37979)
0620 19:01:56.083811 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.083814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.084247 (+ 433us) server_negotiation.cc:407] Connection header received
0620 19:01:56.084391 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.084398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.084472 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.084540 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:01:56.085312 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.085799 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.086789 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.087130 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.087238 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.088767 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:01:56.088771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:01:56.088773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:01:56.088797 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:01:56.090463 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:56.090976 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:56.090980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:56.090982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:56.091066 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:01:56.091417 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:01:56.091421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:01:56.091423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:01:56.091624 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:01:56.091771 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.092140 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.092263 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:01:56.098585 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:01:56.104498 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:01:56.130178 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.114113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34433 (local address 127.0.0.1:47504)
0620 19:01:56.116410 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.116453 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.116572 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.120226 (+ 3654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.120232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.120267 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:56.125627 (+ 5360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.125635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.127101 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.127104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.128325 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.128335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.129569 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.129574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.129711 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.129758 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.129883 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.130005 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2128,"mutex_wait_us":5223,"spinlock_wait_cycles":7040,"thread_start_us":80,"threads_started":1}
I20260620 19:01:56.130853 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.114575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.2.208.130:44943)
0620 19:01:56.125198 (+ 10623us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.125209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.125233 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:01:56.125304 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.125309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.125502 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.125652 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:56.126127 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.127211 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.128650 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.129378 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.129458 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.129560 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.130605 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.130719 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10509,"thread_start_us":89,"threads_started":1}
I20260620 19:01:56.131151 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.114237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.2.208.129:34433)
0620 19:01:56.119567 (+ 5330us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.119576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.119606 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:01:56.119684 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.119690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.119908 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.120046 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:56.125763 (+ 5717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.126981 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.128456 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.129250 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.129354 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.129509 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.130756 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.130841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5224,"thread_start_us":116,"threads_started":1}
I20260620 19:01:56.131449 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.114118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44943 (local address 127.0.0.1:56738)
0620 19:01:56.116969 (+ 2851us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.116982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.117066 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.125883 (+ 8817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.125885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.125903 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:56.125993 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.125999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.127383 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.127386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.128511 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.128519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.129638 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.129641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.130445 (+ 804us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.130466 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.131041 (+ 575us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.131111 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2797,"mutex_wait_us":568,"thread_start_us":92,"threads_started":1}
I20260620 19:01:56.132516 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.114817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44669 (local address 127.0.0.1:44296)
0620 19:01:56.119567 (+ 4750us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.119581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.119683 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.120233 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.120235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.120266 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:56.120416 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.120423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.122475 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.122478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.123925 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.123935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.132162 (+ 8227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.132166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.132247 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.132268 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.132336 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.132398 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4699,"mutex_wait_us":18,"spinlock_wait_cycles":5120,"thread_start_us":70,"threads_started":1}
I20260620 19:01:56.132959 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.115166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.2.208.131:44669)
0620 19:01:56.116846 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.116858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.119780 (+ 2922us) server_negotiation.cc:407] Connection header received
0620 19:01:56.119851 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.119855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.120004 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.120116 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:56.120541 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.122336 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.124071 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.124870 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.125035 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.131316 (+ 6281us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.132729 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.132830 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":130,"threads_started":1}
I20260620 19:01:56.134320 14282 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5dfdd1829548b4b14b045149a3885b (DEFAULT_TABLE table=test-table [id=49f60d615465495999155481d8ee71c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:56.134372 14420 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5dfdd1829548b4b14b045149a3885b (DEFAULT_TABLE table=test-table [id=49f60d615465495999155481d8ee71c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:56.135476 14282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5dfdd1829548b4b14b045149a3885b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:56.135478 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5dfdd1829548b4b14b045149a3885b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:56.135427 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5dfdd1829548b4b14b045149a3885b (DEFAULT_TABLE table=test-table [id=49f60d615465495999155481d8ee71c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:01:56.136276 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5dfdd1829548b4b14b045149a3885b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:01:56.144982 14641 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Bootstrap starting.
I20260620 19:01:56.145418 14640 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: Bootstrap starting.
I20260620 19:01:56.147691 14640 tablet_bootstrap.cc:654] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:56.148859 14641 tablet_bootstrap.cc:654] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:56.150199 14641 log.cc:824] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:56.150643 14639 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Bootstrap starting.
I20260620 19:01:56.151966 14640 log.cc:824] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:56.152655 14641 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: No bootstrap required, opened a new log
I20260620 19:01:56.152848 14641 ts_tablet_manager.cc:1397] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260620 19:01:56.158727 14641 raft_consensus.cc:348] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.159024 14641 raft_consensus.cc:374] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:56.159088 14641 raft_consensus.cc:729] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583080f4b4f9404db5971c39515c9775, State: Initialized, Role: FOLLOWER
I20260620 19:01:56.159611 14641 consensus_queue.cc:260] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.159828 14639 tablet_bootstrap.cc:654] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Neither blocks nor log segments found. Creating new log.
I20260620 19:01:56.160601 14641 ts_tablet_manager.cc:1428] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260620 19:01:56.160580 14639 log.cc:824] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Log is configured to *not* fsync() on all Append() calls
I20260620 19:01:56.160945 14620 heartbeater.cc:503] Master 127.2.208.190:37979 was elected leader, sending a full tablet report...
I20260620 19:01:56.163955 14640 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: No bootstrap required, opened a new log
I20260620 19:01:56.164176 14640 ts_tablet_manager.cc:1397] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260620 19:01:56.169289 14640 raft_consensus.cc:348] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.169565 14640 raft_consensus.cc:374] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:56.169629 14640 raft_consensus.cc:729] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791bd91319584a25b1b8ea13ba86fd4f, State: Initialized, Role: FOLLOWER
I20260620 19:01:56.171710 14639 tablet_bootstrap.cc:492] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: No bootstrap required, opened a new log
I20260620 19:01:56.171938 14639 ts_tablet_manager.cc:1397] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260620 19:01:56.175977 14639 raft_consensus.cc:348] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.176290 14639 raft_consensus.cc:374] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:01:56.176355 14639 raft_consensus.cc:729] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d9f60d0f1647e7b401838e8f010609, State: Initialized, Role: FOLLOWER
I20260620 19:01:56.176869 14639 consensus_queue.cc:260] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.178145 14639 ts_tablet_manager.cc:1428] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260620 19:01:56.178495 14486 heartbeater.cc:503] Master 127.2.208.190:37979 was elected leader, sending a full tablet report...
I20260620 19:01:56.179991 14640 consensus_queue.cc:260] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.181116 14640 ts_tablet_manager.cc:1428] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260620 19:01:56.181457 14348 heartbeater.cc:503] Master 127.2.208.190:37979 was elected leader, sending a full tablet report...
I20260620 19:01:56.196208 14646 raft_consensus.cc:482] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:01:56.196427 14646 raft_consensus.cc:504] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.199689 14646 leader_election.cc:288] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791bd91319584a25b1b8ea13ba86fd4f (127.2.208.129:34433), 583080f4b4f9404db5971c39515c9775 (127.2.208.131:44669)
I20260620 19:01:56.203138 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.197956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34433 (local address 127.2.208.130:53951)
0620 19:01:56.198258 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.198274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.198361 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.199446 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.199449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.199468 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:56.199577 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.199582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.200527 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.200530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.201704 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.201712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.202706 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.202712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.202817 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.202840 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.202905 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.202997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260620 19:01:56.203706 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.198513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:53951 (local address 127.2.208.129:34433)
0620 19:01:56.198631 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.198636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.198650 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:01:56.198701 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.198705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.198756 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.198827 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:56.199677 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.200382 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.201871 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.202481 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.202598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.202650 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.203486 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.203562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:01:56.204121 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.199260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59437 (local address 127.2.208.131:44669)
0620 19:01:56.199392 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.199396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.200017 (+ 621us) server_negotiation.cc:407] Connection header received
0620 19:01:56.200193 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.200196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.200239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.200280 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:01:56.200638 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.201366 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.202784 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.203425 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.203515 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.203558 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.203924 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.203998 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:01:56.204262 14302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5dfdd1829548b4b14b045149a3885b" candidate_uuid: "d9d9f60d0f1647e7b401838e8f010609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791bd91319584a25b1b8ea13ba86fd4f" is_pre_election: true
I20260620 19:01:56.204520 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.198867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:44669 (local address 127.2.208.130:59437)
0620 19:01:56.199942 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.199954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:01:56.200087 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:01:56.200390 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:01:56.200392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:01:56.200408 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:01:56.200531 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.200536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.201512 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.201515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.202656 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:01:56.202668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.203668 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.203671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:01:56.203771 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.203792 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:01:56.204292 (+ 500us) client_negotiation.cc:240] Negotiation successful
0620 19:01:56.204378 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":121,"threads_started":1}
I20260620 19:01:56.204583 14302 raft_consensus.cc:2454] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d9f60d0f1647e7b401838e8f010609 in term 0.
I20260620 19:01:56.205052 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5dfdd1829548b4b14b045149a3885b" candidate_uuid: "d9d9f60d0f1647e7b401838e8f010609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583080f4b4f9404db5971c39515c9775" is_pre_election: true
I20260620 19:01:56.205327 14574 raft_consensus.cc:2454] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d9f60d0f1647e7b401838e8f010609 in term 0.
I20260620 19:01:56.205446 14373 leader_election.cc:302] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791bd91319584a25b1b8ea13ba86fd4f, d9d9f60d0f1647e7b401838e8f010609; no voters:
I20260620 19:01:56.205722 14646 raft_consensus.cc:2781] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:01:56.205796 14646 raft_consensus.cc:482] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:01:56.205837 14646 raft_consensus.cc:3037] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:56.206862 14646 raft_consensus.cc:504] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.207190 14646 leader_election.cc:288] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [CANDIDATE]: Term 1 election: Requested vote from peers 791bd91319584a25b1b8ea13ba86fd4f (127.2.208.129:34433), 583080f4b4f9404db5971c39515c9775 (127.2.208.131:44669)
I20260620 19:01:56.207800 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5dfdd1829548b4b14b045149a3885b" candidate_uuid: "d9d9f60d0f1647e7b401838e8f010609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583080f4b4f9404db5971c39515c9775"
I20260620 19:01:56.207923 14574 raft_consensus.cc:3037] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:56.209131 14574 raft_consensus.cc:2454] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d9f60d0f1647e7b401838e8f010609 in term 1.
I20260620 19:01:56.209514 14302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5dfdd1829548b4b14b045149a3885b" candidate_uuid: "d9d9f60d0f1647e7b401838e8f010609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791bd91319584a25b1b8ea13ba86fd4f"
I20260620 19:01:56.209645 14302 raft_consensus.cc:3037] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:01:56.210839 14302 raft_consensus.cc:2454] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d9f60d0f1647e7b401838e8f010609 in term 1.
I20260620 19:01:56.211262 14373 leader_election.cc:302] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791bd91319584a25b1b8ea13ba86fd4f, d9d9f60d0f1647e7b401838e8f010609; no voters:
I20260620 19:01:56.211598 14646 raft_consensus.cc:2781] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:01:56.211812 14646 raft_consensus.cc:686] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 1 LEADER]: Becoming Leader. State: Replica: d9d9f60d0f1647e7b401838e8f010609, State: Running, Role: LEADER
I20260620 19:01:56.212152 14646 consensus_queue.cc:237] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } }
I20260620 19:01:56.215214 14136 catalog_manager.cc:5495] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9d9f60d0f1647e7b401838e8f010609 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "d9d9f60d0f1647e7b401838e8f010609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9d9f60d0f1647e7b401838e8f010609" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 } health_report { overall_health: UNKNOWN } } }
W20260620 19:01:56.243269 14487 tablet.cc:2369] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:01:56.247346 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.242834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.2.208.130:44943)
0620 19:01:56.242947 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:01:56.242953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:01:56.242989 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:01:56.243183 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:01:56.243186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:01:56.243242 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:01:56.243348 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:01:56.243653 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.244428 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.245620 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:01:56.245962 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:01:56.246083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:01:56.246401 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:01:56.246920 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:01:56.247046 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:01:56.247152 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:01:56.247213 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:01:56.254873 14574 raft_consensus.cc:1261] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 1 FOLLOWER]: Refusing update from remote peer d9d9f60d0f1647e7b401838e8f010609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:01:56.255005 14302 raft_consensus.cc:1261] T 1d5dfdd1829548b4b14b045149a3885b P 791bd91319584a25b1b8ea13ba86fd4f [term 1 FOLLOWER]: Refusing update from remote peer d9d9f60d0f1647e7b401838e8f010609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:01:56.255774 14649 consensus_queue.cc:1035] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "583080f4b4f9404db5971c39515c9775" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:01:56.255936 14646 consensus_queue.cc:1035] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791bd91319584a25b1b8ea13ba86fd4f" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:01:56.265379 14653 mvcc.cc:205] Tried to move back new op lower bound from 7298998748171935744 to 7298998748007796736. Current Snapshot: MvccSnapshot[applied={T|T < 7298998748171935744}]
I20260620 19:01:56.267854 14652 mvcc.cc:205] Tried to move back new op lower bound from 7298998748171935744 to 7298998748007796736. Current Snapshot: MvccSnapshot[applied={T|T < 7298998748171935744}]
I20260620 19:01:56.271628 14654 mvcc.cc:205] Tried to move back new op lower bound from 7298998748171935744 to 7298998748007796736. Current Snapshot: MvccSnapshot[applied={T|T < 7298998748171935744}]
I20260620 19:01:56.280138 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:01:56.280300 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:01:56.282845 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d274b34f763641018d241d09eee5555b: Sending DeleteTablet for 3 replicas of tablet 1d5dfdd1829548b4b14b045149a3885b
I20260620 19:01:56.283878 14420 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5dfdd1829548b4b14b045149a3885b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:56 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:56738
I20260620 19:01:56.284283 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5dfdd1829548b4b14b045149a3885b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:56 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:44296
I20260620 19:01:56.284868 14282 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5dfdd1829548b4b14b045149a3885b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:01:56 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:47504
I20260620 19:01:56.286105 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 14217
I20260620 19:01:56.291555 14662 tablet_replica.cc:330] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: stopping tablet replica
I20260620 19:01:56.291958 14662 raft_consensus.cc:2229] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:01:56.292385 14662 raft_consensus.cc:2258] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:01:56.294085 14662 ts_tablet_manager.cc:1905] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:01:56.296948 14662 ts_tablet_manager.cc:1918] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:01:56.299598 14661 tablet_replica.cc:330] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: stopping tablet replica
I20260620 19:01:56.299940 14661 raft_consensus.cc:2229] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 1 LEADER]: Raft consensus shutting down.
W20260620 19:01:56.300206 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:34433 (error 108)
I20260620 19:01:56.300410 14661 raft_consensus.cc:2258] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:01:56.301322 14662 log.cc:1196] T 1d5dfdd1829548b4b14b045149a3885b P 583080f4b4f9404db5971c39515c9775: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/1d5dfdd1829548b4b14b045149a3885b
I20260620 19:01:56.301581 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 14356
I20260620 19:01:56.301867 14661 ts_tablet_manager.cc:1905] T 1d5dfdd1829548b4b14b045149a3885b P d9d9f60d0f1647e7b401838e8f010609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:01:56.309161 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 14491
I20260620 19:01:56.313272 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.312658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44943 (local address 127.0.0.1:56756)
0620 19:01:56.313082 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.313161 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.130:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323}
I20260620 19:01:56.313573 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:01:56.312896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34433 (local address 127.0.0.1:47518)
0620 19:01:56.313447 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 19:01:56.313502 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484}
W20260620 19:01:56.313747 14120 catalog_manager.cc:4584] TS 791bd91319584a25b1b8ea13ba86fd4f (127.2.208.129:34433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d5dfdd1829548b4b14b045149a3885b: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34433: connect: Connection refused (error 111)
I20260620 19:01:56.315886 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 14104
I20260620 19:01:56.322072 2882 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:01:56.655982 2882 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:01:56.656101 2882 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:01:57.000897 2882 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:01:57.001 UTC [13690] LOG: received smart shutdown request
2026-06-20 19:01:57.004 UTC [13690] LOG: background worker "logical replication launcher" (PID 13699) exited with exit code 1
2026-06-20 19:01:57.004 UTC [13694] LOG: shutting down
2026-06-20 19:01:57.147 UTC [13690] LOG: database system is shut down
2026-06-20T19:01:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43078 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:01:57 dist-test-slave-tsm4 krb5kdc[14673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:01:57 dist-test-slave-tsm4 krb5kdc[14673](info): set up 2 sockets
Jun 20 19:01:57 dist-test-slave-tsm4 krb5kdc[14673](info): commencing operation
krb5kdc: starting...
W20260620 19:01:58.283381 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:01:58 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:01:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangeradmin_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangeradmin_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangerlookup_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangerlookup_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
I20260620 19:01:58.439769 2882 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDDC2sW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/postgres -l logfile start
2026-06-20 19:02:00.746 UTC [14703] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:02:00.746 UTC [14703] LOG: listening on IPv4 address "127.2.208.148", port 48785
2026-06-20 19:02:00.749 UTC [14703] LOG: listening on Unix socket "/tmp/.s.PGSQL.48785"
2026-06-20 19:02:00.784 UTC [14706] LOG: database system was shut down at 2026-06-20 19:01:59 UTC
2026-06-20 19:02:00.788 UTC [14703] LOG: database system is ready to accept connections
I20260620 19:02:01.727100 2882 mini_postgres.cc:96] Postgres bound to 48785
2026-06-20 19:02:01.733 UTC [14714] FATAL: database "slave" does not exist
127.2.208.148:48785 - accepting connections
I20260620 19:02:01.734221 2882 mini_ranger.cc:161] Starting Ranger...
I20260620 19:02:01.748581 2882 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:02:02.174856 2882 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:02:02.174938 2882 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin
2026-06-20 19:02:02,445 [I] DB FLAVOR :POSTGRES
2026-06-20 19:02:02,445 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:02:02,445 [I] Checking connection..
2026-06-20 19:02:02,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:02:02,682 [I] Checking connection passed.
2026-06-20 19:02:02,682 [I] --------- Verifying version history table ---------
2026-06-20 19:02:02,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:02:02,921 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:02:02,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:02:03,158 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:02:03,158 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:02:03,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:02:03.383 UTC [14778] WARNING: there is no transaction in progress
2026-06-20 19:02:03,389 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:02:03,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:02:03,620 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:02:03,620 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:02:03,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:02:03,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:02:04,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tsm4.c.gcp-upstream.internal','N') ;"
2026-06-20 19:02:04,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:02:04,603 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:02:04,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:02:04,841 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:02:04,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:02:05,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:02:05,333 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:02:05,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:02:06.248 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.263 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.278 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.292 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.307 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.323 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.374 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.381 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.395 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.404 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.414 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.426 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.435 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.441 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.447 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:06.453 UTC [14886] WARNING: there is no transaction in progress
2026-06-20 19:02:07,034 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:02:07,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:02:07,262 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:02:07,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:02:07,519 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:02:07,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:02:07,776 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:02:07,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:02:08,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:02:08,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tsm4.c.gcp-upstream.internal';"
2026-06-20 19:02:08,512 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:02:08,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:02:08,751 [I] DB_PATCHES have already been applied
I20260620 19:02:08.757368 2882 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDDC2sW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:02:08.757431 2882 mini_ranger.cc:193] Using host: 127.2.208.148
I20260620 19:02:08.760888 2882 mini_ranger.cc:227] Ranger admin URL: http://127.2.208.148:55401
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:02:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:02:09.865118 2882 mini_ranger.cc:160] Time spent starting Ranger: real 8.131s user 0.004s sys 0.005s
Jun 20, 2026 7:02:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:02:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:02:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:02:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:02:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:02:29.870366 2882 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.208.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.208.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangerkms_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.208.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/rangerkms_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.208.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.208.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.208.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/HTTP_127.2.208.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:02:29 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982149, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:02:29.950186 2882 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:02:29.959419 2882 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:02:30.760746 2882 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:02:30.760831 2882 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms
I20260620 19:02:30.760854 2882 mini_ranger_kms.cc:227] Using postgres at 127.2.208.148:48785
2026-06-20 19:02:30,925 [I] DB FLAVOR :POSTGRES
2026-06-20 19:02:30,925 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:02:30,925 [I] Checking connection
2026-06-20 19:02:30,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:02:31,193 [I] connection success
2026-06-20 19:02:31,193 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:02:31,193 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:02:31,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:02:31,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:02:31,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:02:31,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:02:31,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.208.148:48785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:02:31,714 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:02:32.177889 2882 mini_ranger_kms.cc:310] Created kms service
I20260620 19:02:32.387096 2882 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:02:32.429427 2882 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:02:32 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982149, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:32.803432 2882 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:02:32.803588 2882 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDDC2sW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDDC2sW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:02:32.803610 2882 mini_ranger_kms.cc:240] Using host: 127.2.208.148
I20260620 19:02:32.807225 2882 mini_ranger_kms.cc:276] Ranger KMS PID: 15066
I20260620 19:02:32.807307 2882 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.208.148:34493
Jun 20, 2026 7:02:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:02:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:02:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:02:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:02:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:02:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 19:02:34 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:35.065668 2882 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.115s user 0.013s sys 0.136s
I20260620 19:02:35.065822 2882 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:02:35.065915 2882 mini_ranger_kms.cc:401] 127.2.208.148:34493/kms/v1/keys
Jun 20 19:02:35 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
Jun 20 19:02:35 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.208.148@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
Jun 20 19:02:37 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:02:37.874037 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43803
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43803
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:37.981446 15117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:37.981720 15117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:37.981792 15117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:37.985770 15117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:37.985849 15117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:37.985888 15117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:37.985941 15117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:37.985976 15117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:37.992506 15117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.208.148:34493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43803
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:37.993553 15117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:37.994524 15117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:38.002996 15125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:38.003147 15117 server_base.cc:1034] running on GCE node
W20260620 19:02:38.003203 15123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:38.003026 15122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:38.003860 15117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:38.004844 15117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:38.006022 15117 hybrid_clock.cc:648] HybridClock initialized: now 1781982158006010 us; error 37 us; skew 500 ppm
Jun 20 19:02:38 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982158, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:38.011725 15117 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:38.012682 15117 webserver.cc:460] Webserver started at http://127.2.208.190:33663/ using document root <none> and password file <none>
I20260620 19:02:38.013214 15117 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:38.013290 15117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:38.013518 15117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:02:38 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982158, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:38.061034 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "99467dcc34ef4297a06040763f107081"
format_stamp: "Formatted at 2026-06-20 19:02:38 on dist-test-slave-tsm4"
server_key: "c983aec563049b13599979d6dafed674"
server_key_iv: "f8710e4436357940f513f2b916562c5d"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:38.061678 15117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "99467dcc34ef4297a06040763f107081"
format_stamp: "Formatted at 2026-06-20 19:02:38 on dist-test-slave-tsm4"
server_key: "c983aec563049b13599979d6dafed674"
server_key_iv: "f8710e4436357940f513f2b916562c5d"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:38.065891 15117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:02:38.079241 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:38.080757 15117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:02:38.080889 15117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "99467dcc34ef4297a06040763f107081"
format_stamp: "Formatted at 2026-06-20 19:02:38 on dist-test-slave-tsm4"
server_key: "c983aec563049b13599979d6dafed674"
server_key_iv: "f8710e4436357940f513f2b916562c5d"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:38.081022 15117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:38.100172 15117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:38.103215 15117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:38.103436 15117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:38.110391 15117 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:43803
I20260620 19:02:38.110430 15184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:43803 every 8 connection(s)
I20260620 19:02:38.111554 15117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:38.114955 15185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:38.120925 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15117
I20260620 19:02:38.121053 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:02:38.121335 2882 external_mini_cluster.cc:1428] Setting key e3a984ef492eb13973b353fcf0d4fc5e
I20260620 19:02:38.121665 15185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Bootstrap starting.
I20260620 19:02:38.124379 15185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:38.125263 15185 log.cc:824] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:38.127358 15185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: No bootstrap required, opened a new log
Jun 20 19:02:38 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:38.131704 15185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:38.131958 15185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:38.132046 15185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99467dcc34ef4297a06040763f107081, State: Initialized, Role: FOLLOWER
I20260620 19:02:38.132500 15185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:38.132642 15185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:38.132723 15185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:38.132817 15185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:38.133841 15185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:38.134189 15185 leader_election.cc:302] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99467dcc34ef4297a06040763f107081; no voters:
I20260620 19:02:38.134542 15185 leader_election.cc:288] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:02:38.134624 15190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:38.135106 15190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 LEADER]: Becoming Leader. State: Replica: 99467dcc34ef4297a06040763f107081, State: Running, Role: LEADER
I20260620 19:02:38.135483 15190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:38.135764 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:38.122741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.2.208.190:43803)
0620 19:02:38.123146 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:38.123162 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:38.123195 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:02:38.123851 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:38.123867 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:38.124192 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:38.124528 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:38.125291 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:38.126492 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:38.127333 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:38.127971 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:38.128140 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:38.130571 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:38.130582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:38.130595 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:38.130619 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:38.132416 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:38.132899 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:38.132905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:38.132910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:38.133024 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:38.133309 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:38.133314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:38.133315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:38.133693 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:38.133890 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:38.134472 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 19:02:38.134755 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260620 19:02:38.136085 15185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:38.136709 15191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99467dcc34ef4297a06040763f107081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } } }
I20260620 19:02:38.136934 15191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:38.136715 15190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99467dcc34ef4297a06040763f107081. Latest consensus state: current_term: 1 leader_uuid: "99467dcc34ef4297a06040763f107081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } } }
I20260620 19:02:38.137182 15190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:38.138259 15185 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:02:38.138957 15199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:38.146158 15199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:38.156903 15199 catalog_manager.cc:1347] Generated new cluster ID: e1185a373bd940d190d0aa1331f167d9
I20260620 19:02:38.156966 15199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:38.178799 15199 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:02:38.180131 15199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:38.185195 15199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Generated new TSK 0
I20260620 19:02:38.186029 15199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:02:38.870640 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:39.009349 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:39.009799 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:39.009929 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:39.016232 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:39.016310 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:39.016402 15230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:02:39.024736 15230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:39.026127 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:39.027525 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:39.039469 15237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:39.039711 15240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:39.039888 15230 server_base.cc:1034] running on GCE node
W20260620 19:02:39.039570 15238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:39.040498 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:39.041249 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:39.042459 15230 hybrid_clock.cc:648] HybridClock initialized: now 1781982159042400 us; error 63 us; skew 500 ppm
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:39.045724 15230 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:02:39.046634 15230 webserver.cc:460] Webserver started at http://127.2.208.129:33813/ using document root <none> and password file <none>
I20260620 19:02:39.047219 15230 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:39.047317 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:39.047573 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:39.064344 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "de141ac2eb244611ace6bad80788eee6"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "4222d98f26c8222e847df9ac4beb37bb"
server_key_iv: "3b34faea82fdc9d75b6e13944d5f6fc9"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.066394 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "de141ac2eb244611ace6bad80788eee6"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "4222d98f26c8222e847df9ac4beb37bb"
server_key_iv: "3b34faea82fdc9d75b6e13944d5f6fc9"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.070899 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:02:39.093080 15247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.094760 15230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:02:39.094878 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "de141ac2eb244611ace6bad80788eee6"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "4222d98f26c8222e847df9ac4beb37bb"
server_key_iv: "3b34faea82fdc9d75b6e13944d5f6fc9"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.095000 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:39.113078 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:39.116480 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:39.116748 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:39.117444 15230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:39.118559 15230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:39.118665 15230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.118786 15230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:39.118865 15230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.131191 15230 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:39889
I20260620 19:02:39.132436 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:02:39.138357 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15230
I20260620 19:02:39.138471 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:02:39.138779 2882 external_mini_cluster.cc:1428] Setting key 6808f3a50ce20804ae57d38661c11d91
I20260620 19:02:39.148610 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:39889 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:39.177039 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.152238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:41641 (local address 127.2.208.190:43803)
0620 19:02:39.158995 (+ 6757us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:39.159005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:39.163286 (+ 4281us) server_negotiation.cc:407] Connection header received
0620 19:02:39.163349 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:39.163352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:39.163432 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:39.163668 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:39.165404 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.166364 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.168646 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.169027 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.169113 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.172376 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:39.172380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:39.172384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:39.172414 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:39.174344 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.175018 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.175022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.175024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.175135 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.175607 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.175611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.175613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.175878 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:39.175988 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:39.176690 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 19:02:39.176826 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6647,"thread_start_us":86,"threads_started":1}
I20260620 19:02:39.178684 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.151079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.129:41641)
0620 19:02:39.159330 (+ 8251us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:39.159431 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:39.160530 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:39.164069 (+ 3539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:39.164081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:39.164468 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:39.165212 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.165234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.166925 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.166929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.168467 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.168479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.169338 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.169342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.169549 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.170268 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:39.170289 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:39.172134 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:39.174482 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.174488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.174505 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.174876 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.175248 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.175251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.175253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.175474 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.176062 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:39.176069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:39.176392 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 19:02:39.177236 (+ 844us) client_negotiation.cc:240] Negotiation successful
0620 19:02:39.177577 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7992,"thread_start_us":185,"threads_started":1}
I20260620 19:02:39.180351 15363 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:39.180603 15363 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:39.181210 15363 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:39.183043 15149 ts_manager.cc:194] Registered new tserver with Master: de141ac2eb244611ace6bad80788eee6 (127.2.208.129:39889)
I20260620 19:02:39.184669 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:41641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:02:39.200165 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:02:39.358521 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:39.358862 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:39.358927 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:39.365005 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:39.365091 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:39.365178 15369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:02:39.372843 15369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:39.374329 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:39.375607 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:39.387240 15375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:39.387284 15376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:39.387341 15378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:39.391289 15369 server_base.cc:1034] running on GCE node
I20260620 19:02:39.391855 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:39.392638 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:39.393836 15369 hybrid_clock.cc:648] HybridClock initialized: now 1781982159393805 us; error 44 us; skew 500 ppm
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:39.397571 15369 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:02:39.398679 15369 webserver.cc:460] Webserver started at http://127.2.208.130:36825/ using document root <none> and password file <none>
I20260620 19:02:39.399428 15369 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:39.399681 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:39.400046 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:39.417351 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "a0df86dcfc9b4e9daca046fd84036c39"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "c6127843acec8b06074fa3d48d50e1ba"
server_key_iv: "d0c7c87232ed3d908beee63edfd6e0c0"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.418349 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "a0df86dcfc9b4e9daca046fd84036c39"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "c6127843acec8b06074fa3d48d50e1ba"
server_key_iv: "d0c7c87232ed3d908beee63edfd6e0c0"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.423693 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 19:02:39.442793 15385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.444515 15369 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 19:02:39.444644 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "a0df86dcfc9b4e9daca046fd84036c39"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "c6127843acec8b06074fa3d48d50e1ba"
server_key_iv: "d0c7c87232ed3d908beee63edfd6e0c0"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.444772 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:39.475091 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:39.478863 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:39.479058 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:39.479806 15369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:39.480873 15369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:39.481009 15369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.481119 15369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:39.481176 15369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.492468 15369 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:34715
I20260620 19:02:39.493603 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:02:39.494414 15498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:34715 every 8 connection(s)
I20260620 19:02:39.498950 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15369
I20260620 19:02:39.499074 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:02:39.499563 2882 external_mini_cluster.cc:1428] Setting key ec38526986c6a12c2d6589fea77acb90
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:39.509142 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.495840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:33539 (local address 127.2.208.190:43803)
0620 19:02:39.495978 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:39.495983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:39.497145 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 19:02:39.498385 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:39.498388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:39.498446 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:39.498601 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:39.500963 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.501650 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.502761 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.503084 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.503192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.505544 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:39.505548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:39.505551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:39.505578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:39.507051 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.507608 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.507611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.507613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.507696 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.508046 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.508049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.508050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.508226 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:39.508334 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:39.508843 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 19:02:39.508967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:02:39.509980 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.495997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.130:33539)
0620 19:02:39.496953 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:39.497010 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:39.498080 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:39.498791 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:39.498800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:39.499163 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:39.500774 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.500805 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.501779 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.501781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.502631 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.502638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.503233 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.503236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.503444 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.504051 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:39.504067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:39.505380 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:39.507160 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.507164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.507175 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.507491 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.507795 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.507798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.507800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.507932 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.508329 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:39.508335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:39.508580 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 19:02:39.508787 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 19:02:39.509073 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":123,"threads_started":1}
I20260620 19:02:39.511206 15499 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:39.511484 15499 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:39.512036 15499 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:39.513166 15149 ts_manager.cc:194] Registered new tserver with Master: a0df86dcfc9b4e9daca046fd84036c39 (127.2.208.130:34715)
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:02:39.513877 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:33539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:02:39.534576 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:39.666967 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:39.667356 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:39.667480 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:39.673257 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:39.673373 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:39.673484 15504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:02:39.681160 15504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:39.682576 15504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:39.683897 15504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
W20260620 19:02:39.696223 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:39.696628 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:39.696856 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:39.697928 15504 server_base.cc:1034] running on GCE node
I20260620 19:02:39.698355 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:39.699051 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:39.700238 15504 hybrid_clock.cc:648] HybridClock initialized: now 1781982159700183 us; error 85 us; skew 500 ppm
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:39.703946 15504 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:02:39.704985 15504 webserver.cc:460] Webserver started at http://127.2.208.131:40403/ using document root <none> and password file <none>
I20260620 19:02:39.705705 15504 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:39.705838 15504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:39.706112 15504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:39.741605 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "d340c53d4c484b5ea16adea8ee9b3995"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "2f8a72adc52eb69ee9943c7aff9f6bda"
server_key_iv: "237efe6a283fc31e6c552b6ae877b454"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.742420 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "d340c53d4c484b5ea16adea8ee9b3995"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "2f8a72adc52eb69ee9943c7aff9f6bda"
server_key_iv: "237efe6a283fc31e6c552b6ae877b454"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.754654 15504 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.000s
I20260620 19:02:39.781263 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.783130 15504 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.000s
I20260620 19:02:39.783340 15504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "d340c53d4c484b5ea16adea8ee9b3995"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "2f8a72adc52eb69ee9943c7aff9f6bda"
server_key_iv: "237efe6a283fc31e6c552b6ae877b454"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:39.783588 15504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:39.810153 15504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:39.813055 15504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:39.813235 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:39.813771 15504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:39.814606 15504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:39.814646 15504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.814728 15504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:39.814762 15504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:39.825168 15504 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:33799
I20260620 19:02:39.825212 15632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:33799 every 8 connection(s)
I20260620 19:02:39.826190 15504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:02:39.835127 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15504
I20260620 19:02:39.835227 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:02:39.835547 2882 external_mini_cluster.cc:1428] Setting key 05a05887ef049cb4c3be1650d5b541f0
Jun 20 19:02:39 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:39.842432 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.828589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:38855 (local address 127.2.208.190:43803)
0620 19:02:39.830005 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:39.830009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:39.830036 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:02:39.831276 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:39.831280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:39.831331 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:39.831449 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:39.832711 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.833465 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.835090 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.835422 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.835513 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.838504 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:39.838508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:39.838511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:39.838537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:39.840227 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.840764 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.840767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.840769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.840854 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:39.841255 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:39.841258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:39.841260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:39.841510 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:39.841606 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:39.842173 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 19:02:39.842279 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327}
I20260620 19:02:39.843796 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:39.829042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.131:38855)
0620 19:02:39.829581 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:39.829640 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:39.831047 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:39.831629 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:39.831638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:39.831959 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:39.832560 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.832577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.833613 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.833617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.834968 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:39.834980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:39.835766 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:39.835770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:39.836055 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:39.836663 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:39.836682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:39.838343 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:39.840350 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.840355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.840370 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.840656 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.840956 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:39.840959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:39.840961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:39.841118 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:39.841625 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:39.841631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:39.841906 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 19:02:39.842572 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 19:02:39.842853 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260620 19:02:39.844931 15633 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:39.845166 15633 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:39.845671 15633 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:39.846720 15149 ts_manager.cc:194] Registered new tserver with Master: d340c53d4c484b5ea16adea8ee9b3995 (127.2.208.131:33799)
I20260620 19:02:39.847487 15149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:38855
I20260620 19:02:39.850911 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:02:39.851704 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15230
I20260620 19:02:39.859094 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15369
I20260620 19:02:39.867153 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15504
I20260620 19:02:39.873518 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15117
I20260620 19:02:39.881503 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43803
--webserver_interface=127.2.208.190
--webserver_port=33663
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43803
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:39.989604 15636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:39.989845 15636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:39.989881 15636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:39.993854 15636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:39.993901 15636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:39.993919 15636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:39.993935 15636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:39.993984 15636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:40.000439 15636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43803
--ranger_config_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:40.001365 15636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:40.002300 15636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:40.010320 15642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.010376 15641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.010378 15644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:40.010409 15636 server_base.cc:1034] running on GCE node
I20260620 19:02:40.011046 15636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:40.012044 15636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:40.013198 15636 hybrid_clock.cc:648] HybridClock initialized: now 1781982160013186 us; error 25 us; skew 500 ppm
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:40.015843 15636 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:40.016647 15636 webserver.cc:460] Webserver started at http://127.2.208.190:33663/ using document root <none> and password file <none>
I20260620 19:02:40.017166 15636 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:40.017230 15636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:40.060547 15636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:02:40.062712 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.064049 15636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:40.064189 15636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "99467dcc34ef4297a06040763f107081"
format_stamp: "Formatted at 2026-06-20 19:02:38 on dist-test-slave-tsm4"
server_key: "c983aec563049b13599979d6dafed674"
server_key_iv: "f8710e4436357940f513f2b916562c5d"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:40.064718 15636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:40.083963 15636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:40.086792 15636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:40.086990 15636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:40.093961 15636 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:43803
I20260620 19:02:40.093966 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:43803 every 8 connection(s)
I20260620 19:02:40.095045 15636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:40.099025 15705 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:02:40.099550 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15636
I20260620 19:02:40.099965 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:39889
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:43803
--webserver_port=33813
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:40.100761 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Bootstrap starting.
I20260620 19:02:40.115978 15705 log.cc:824] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:40.121847 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:02:40.122202 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Bootstrap complete.
I20260620 19:02:40.126507 15705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:40.126916 15705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99467dcc34ef4297a06040763f107081, State: Initialized, Role: FOLLOWER
I20260620 19:02:40.127493 15705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:40.127636 15705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:40.127702 15705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:40.127794 15705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:02:40.129576 15705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:40.129899 15705 leader_election.cc:302] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99467dcc34ef4297a06040763f107081; no voters:
I20260620 19:02:40.130187 15705 leader_election.cc:288] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:02:40.130311 15709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:02:40.130563 15709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [term 2 LEADER]: Becoming Leader. State: Replica: 99467dcc34ef4297a06040763f107081, State: Running, Role: LEADER
I20260620 19:02:40.130910 15709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } }
I20260620 19:02:40.131425 15705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:40.131930 15710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99467dcc34ef4297a06040763f107081. Latest consensus state: current_term: 2 leader_uuid: "99467dcc34ef4297a06040763f107081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } } }
I20260620 19:02:40.132050 15710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:40.132360 15711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "99467dcc34ef4297a06040763f107081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99467dcc34ef4297a06040763f107081" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43803 } } }
I20260620 19:02:40.132457 15711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081 [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:40.132776 15713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:40.135802 15713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:40.136348 15705 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:02:40.136407 15713 catalog_manager.cc:1259] Loaded cluster ID: e1185a373bd940d190d0aa1331f167d9
I20260620 19:02:40.136454 15713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:40.138732 15713 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:40.139389 15713 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Loaded TSK: 0
I20260620 19:02:40.142406 15713 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:02:40.239389 15707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:40.239698 15707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:40.239760 15707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:40.245893 15707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:40.246017 15707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:40.246124 15707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:02:40.253836 15707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:39889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=33813
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:40.255074 15707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:40.256505 15707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:40.268651 15745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.268828 15743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.268707 15742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:40.269328 15707 server_base.cc:1034] running on GCE node
I20260620 19:02:40.269698 15707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:40.270244 15707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:40.271432 15707 hybrid_clock.cc:648] HybridClock initialized: now 1781982160271401 us; error 42 us; skew 500 ppm
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:40.274339 15707 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:02:40.275246 15707 webserver.cc:460] Webserver started at http://127.2.208.129:33813/ using document root <none> and password file <none>
I20260620 19:02:40.275923 15707 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:40.276054 15707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:40.293251 15707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:02:40.295939 15753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.297536 15707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:02:40.297694 15707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "de141ac2eb244611ace6bad80788eee6"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "4222d98f26c8222e847df9ac4beb37bb"
server_key_iv: "3b34faea82fdc9d75b6e13944d5f6fc9"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:40.298441 15707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:40.315258 15707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:40.318856 15707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:40.319063 15707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:40.319705 15707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:40.320617 15707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:40.320685 15707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.320756 15707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:40.320799 15707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.331560 15707 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:39889
I20260620 19:02:40.331614 15866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:39889 every 8 connection(s)
I20260620 19:02:40.332955 15707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:02:40.337772 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15707
I20260620 19:02:40.338295 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:34715
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:43803
--webserver_port=36825
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:40.355933 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.335097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.129:46871)
0620 19:02:40.338203 (+ 3106us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:40.338259 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:40.339256 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:40.341343 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:40.341349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:40.341593 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:40.342012 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.342025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.344115 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.344122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.345727 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.345740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.347192 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.347198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.347444 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.348202 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:40.348228 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:40.350071 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:40.352641 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.352646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.352659 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.352968 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.353347 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.353350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.353352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.353521 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.354220 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:40.354225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:40.354478 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 19:02:40.354654 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 19:02:40.354932 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2876,"thread_start_us":132,"threads_started":1}
I20260620 19:02:40.357359 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.334983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:46871 (local address 127.2.208.190:43803)
0620 19:02:40.335507 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:40.335530 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:40.339565 (+ 4035us) server_negotiation.cc:407] Connection header received
0620 19:02:40.340094 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:40.340111 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:40.340544 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:40.340905 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:40.342173 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.343914 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.345899 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.346733 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.346930 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.350304 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:40.350312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:40.350331 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:40.350366 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:40.352509 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.353110 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.353116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.353121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.353249 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.353617 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.353620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.353621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.354051 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:40.354168 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:40.356272 (+ 2104us) server_negotiation.cc:299] Negotiation successful
0620 19:02:40.356533 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":115,"threads_started":1}
I20260620 19:02:40.360384 15867 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:40.360615 15867 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.361168 15867 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.363279 15867 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:43803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:02:40.363423 15867 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.363571 15867 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:40.364445 15867 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.364595 15867 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.365361 15867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:02:40.479053 15871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:40.479465 15871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:40.479557 15871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:40.484660 15871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:40.484732 15871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:40.484867 15871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:02:40.490435 15871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:34715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=36825
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:40.491562 15871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:40.492625 15871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:40.503249 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.503278 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.503518 15879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:40.503804 15871 server_base.cc:1034] running on GCE node
I20260620 19:02:40.504220 15871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:40.504907 15871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:40.506067 15871 hybrid_clock.cc:648] HybridClock initialized: now 1781982160506045 us; error 32 us; skew 500 ppm
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:40.509629 15871 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:02:40.510685 15871 webserver.cc:460] Webserver started at http://127.2.208.130:36825/ using document root <none> and password file <none>
I20260620 19:02:40.511430 15871 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:40.511581 15871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:40.528580 15871 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:02:40.531097 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.532883 15871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:02:40.533004 15871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "a0df86dcfc9b4e9daca046fd84036c39"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "c6127843acec8b06074fa3d48d50e1ba"
server_key_iv: "d0c7c87232ed3d908beee63edfd6e0c0"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:40.533687 15871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:40.564232 15871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:40.568061 15871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:40.568346 15871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:40.569099 15871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:40.570166 15871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:40.570233 15871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.570289 15871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:40.570313 15871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.581806 15871 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:34715
I20260620 19:02:40.581945 16000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:34715 every 8 connection(s)
I20260620 19:02:40.582860 15871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:02:40.587724 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 15871
I20260620 19:02:40.588181 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.208.148:34493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:33799
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:43803
--webserver_port=40403
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:40.601411 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.584854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:47433 (local address 127.2.208.190:43803)
0620 19:02:40.584979 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:40.584984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:40.585888 (+ 904us) server_negotiation.cc:407] Connection header received
0620 19:02:40.586901 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:40.586905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:40.586967 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:40.587051 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:40.588265 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.588875 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.590494 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.590840 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.590913 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.593922 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:40.593927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:40.593931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:40.593964 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:40.595851 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.599263 (+ 3412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.599269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.599274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.599390 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.599879 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.599884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.599888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.600145 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:40.600231 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:40.601020 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 19:02:40.601168 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:02:40.602948 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.585287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.130:47433)
0620 19:02:40.585743 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:40.585792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:40.586680 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:40.587216 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:40.587223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:40.587604 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:40.588122 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.588136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.589001 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.589005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.590340 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.590355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.591097 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.591101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.591243 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.591943 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:40.591967 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:40.593708 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:40.598739 (+ 5031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.598746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.598762 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.599132 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.599519 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.599524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.599528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.599750 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.600325 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:40.600333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:40.600709 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 19:02:40.601666 (+ 957us) client_negotiation.cc:240] Negotiation successful
0620 19:02:40.601976 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":135,"threads_started":1}
I20260620 19:02:40.604409 16001 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:40.604632 16001 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.605125 16001 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.606623 16001 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:43803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:02:40.606729 16001 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.606863 16001 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:40.607642 16001 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.607785 16001 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.608446 16001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:02:40.718254 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:40.718515 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:40.718616 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:40.722990 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:40.723081 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:40.723184 16004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:02:40.728981 16004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.208.148:34493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:33799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=40403
--tserver_master_addrs=127.2.208.190:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:40.730187 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:40.731513 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:40.743805 16010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:40.743816 16011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:40.744442 16004 server_base.cc:1034] running on GCE node
W20260620 19:02:40.747083 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:40.747548 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:40.748135 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:40.749313 16004 hybrid_clock.cc:648] HybridClock initialized: now 1781982160749288 us; error 47 us; skew 500 ppm
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:40.752009 16004 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:02:40.752995 16004 webserver.cc:460] Webserver started at http://127.2.208.131:40403/ using document root <none> and password file <none>
I20260620 19:02:40.753530 16004 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:40.753609 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:40.769970 16004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:02:40.773093 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.774663 16004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:02:40.774858 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "d340c53d4c484b5ea16adea8ee9b3995"
format_stamp: "Formatted at 2026-06-20 19:02:39 on dist-test-slave-tsm4"
server_key: "2f8a72adc52eb69ee9943c7aff9f6bda"
server_key_iv: "237efe6a283fc31e6c552b6ae877b454"
server_key_version: "kuduclusterkey@0"
I20260620 19:02:40.775683 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:40.798593 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:40.802296 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:40.802502 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:40.803227 16004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:40.804374 16004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:40.804442 16004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.804497 16004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:40.804534 16004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:40.815956 16004 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:33799
I20260620 19:02:40.815976 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:33799 every 8 connection(s)
I20260620 19:02:40.817109 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:02:40.820730 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16004
Jun 20 19:02:40 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:40.839184 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.819060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:47765 (local address 127.2.208.190:43803)
0620 19:02:40.819229 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:40.819235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:40.820298 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 19:02:40.821633 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:40.821637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:40.821698 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:40.821756 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:40.823393 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.824183 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.825979 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.826343 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.826444 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.831233 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:40.831236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:40.831241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:40.831268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:40.836456 (+ 5188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.837076 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.837081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.837084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.837202 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.837905 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.837910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.837912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.838151 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:40.838236 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:40.838834 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 19:02:40.838995 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 19:02:40.840178 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.821790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.2.208.190:43803)
0620 19:02:40.822000 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:40.822004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:40.823242 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 19:02:40.824315 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:40.824318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:40.824365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:40.824458 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:40.828703 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.829424 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.830678 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.830899 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.831051 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.836469 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:40.836473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:40.836478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:40.836506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:40.838111 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.838661 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.838665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.838666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.838752 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:40.839046 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:40.839050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:40.839051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:40.839243 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:40.839526 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:40.839881 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 19:02:40.839988 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260620 19:02:40.841017 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:40.819511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43803 (local address 127.2.208.131:47765)
0620 19:02:40.820120 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:40.820198 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:40.821359 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:40.822169 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:40.822178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:40.822517 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:40.823168 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.823189 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.824374 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.824378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.825827 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:40.825840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:40.826664 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:40.826668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:40.826879 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:40.828545 (+ 1666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:40.828581 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:40.830977 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:40.836582 (+ 5605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.836588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.836605 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.836966 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.837593 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:40.837596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:40.837598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:40.837789 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:40.838294 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:40.838299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:40.838555 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 19:02:40.839353 (+ 798us) client_negotiation.cc:240] Negotiation successful
0620 19:02:40.839835 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":139,"threads_started":1}
I20260620 19:02:40.842444 16135 heartbeater.cc:347] Connected to a master server at 127.2.208.190:43803
I20260620 19:02:40.842674 16135 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.843181 16135 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.844779 16135 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:43803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:02:40.844910 16135 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.845077 16135 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:40.845863 16135 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:40.846014 16135 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
W20260620 19:02:40.846774 16135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:02:41 dist-test-slave-tsm4 krb5kdc[14673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982161, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:41.365574 15867 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:41.365981 15867 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:41.367134 15669 ts_manager.cc:194] Registered new tserver with Master: de141ac2eb244611ace6bad80788eee6 (127.2.208.129:39889)
I20260620 19:02:41.368852 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:46871
I20260620 19:02:41.608654 16001 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:41.608944 16001 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:41.610148 15668 ts_manager.cc:194] Registered new tserver with Master: a0df86dcfc9b4e9daca046fd84036c39 (127.2.208.130:34715)
I20260620 19:02:41.611116 15668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:47433
Jun 20 19:02:41 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982161, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:41.847007 16135 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:41.847312 16135 heartbeater.cc:511] Master 127.2.208.190:43803 requested a full tablet report, sending...
I20260620 19:02:41.848146 15668 ts_manager.cc:194] Registered new tserver with Master: d340c53d4c484b5ea16adea8ee9b3995 (127.2.208.131:33799)
I20260620 19:02:41.848872 15668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:47765
I20260620 19:02:41.855715 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:02:41.868652 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.857236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.2.208.190:43803)
0620 19:02:41.859496 (+ 2260us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:41.859501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:41.859514 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:02:41.859551 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:41.859554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:41.859594 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:41.859678 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:41.860600 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.861444 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.862574 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.862900 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.862986 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.864650 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:41.864654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:41.864657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:41.864681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:41.866781 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:41.867265 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:41.867269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:41.867271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:41.867378 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:41.867740 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:41.867744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:41.867746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:41.867971 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:41.868108 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:41.868331 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:02:41.868457 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2182,"thread_start_us":92,"threads_started":1}
I20260620 19:02:41.873047 15668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:02:41.877796 15668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 20 19:02:41 dist-test-slave-tsm4 krb5kdc[14673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982161, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for HTTP/127.2.208.148@KRBTEST.COM
I20260620 19:02:41.898367 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.889902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39889 (local address 127.0.0.1:40346)
0620 19:02:41.891802 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:41.891850 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:41.891977 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:41.893165 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:41.893172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:41.893198 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:41.893339 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.893346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.895013 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.895017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.896442 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.896453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.897771 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.897775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.897885 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.897925 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:02:41.898037 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:02:41.898157 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1699,"thread_start_us":81,"threads_started":1}
I20260620 19:02:41.899040 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.890691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.2.208.129:39889)
0620 19:02:41.892650 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:41.892660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:41.892687 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:02:41.892763 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:41.892768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:41.892976 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:41.893137 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:41.893497 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.894865 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.896604 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.897427 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.897545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.897704 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:41.898792 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0620 19:02:41.898906 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1835,"thread_start_us":101,"threads_started":1}
I20260620 19:02:41.899866 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.890711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:33799 (local address 127.0.0.1:39042)
0620 19:02:41.893452 (+ 2741us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:41.893463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:41.893525 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:41.894870 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:41.894873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:41.894889 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:41.894984 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.894990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.896566 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.896570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.898400 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.898412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.899437 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.899441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.899544 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.899569 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:02:41.899646 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:02:41.899713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2687,"thread_start_us":97,"threads_started":1}
I20260620 19:02:41.900432 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.891052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.2.208.131:33799)
0620 19:02:41.893738 (+ 2686us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:41.893744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:41.893766 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:02:41.893834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:41.893839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:41.894020 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:41.894172 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:41.895132 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.896441 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.898535 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.899298 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.899437 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.899572 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:41.900189 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 19:02:41.900296 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2588,"thread_start_us":64,"threads_started":1}
I20260620 19:02:41.904448 16056 tablet_service.cc:1467] Processing CreateTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 (DEFAULT_TABLE table=test-table [id=5e3cce7b8b1b49fda24d5cb4c43eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:41.905599 16056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:41.905520 15791 tablet_service.cc:1467] Processing CreateTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 (DEFAULT_TABLE table=test-table [id=5e3cce7b8b1b49fda24d5cb4c43eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:41.905961 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.890381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:34715 (local address 127.0.0.1:40962)
0620 19:02:41.892198 (+ 1817us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:41.892209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:41.893279 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:41.901217 (+ 7938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:41.901220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:41.901239 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:41.901321 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.901327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.902703 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.902707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.904448 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:41.904459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.905597 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.905601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:41.905696 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.905720 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:02:41.905776 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:02:41.905840 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":105,"threads_started":1}
I20260620 19:02:41.906499 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:41.891518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.2.208.130:34715)
0620 19:02:41.899786 (+ 8268us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:41.899794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:41.899816 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:02:41.899889 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:41.899895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:41.901072 (+ 1177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:41.901223 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:41.901461 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.902566 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.904609 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:41.905438 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:41.906164 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:41.906294 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:41.906333 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 19:02:41.906397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8153,"thread_start_us":107,"threads_started":1}
I20260620 19:02:41.906519 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:41.908787 15935 tablet_service.cc:1467] Processing CreateTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 (DEFAULT_TABLE table=test-table [id=5e3cce7b8b1b49fda24d5cb4c43eb989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:41.909456 15935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:41.917415 16171 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Bootstrap starting.
I20260620 19:02:41.919732 16169 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Bootstrap starting.
I20260620 19:02:41.919876 16171 tablet_bootstrap.cc:654] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:41.920845 16171 log.cc:824] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:41.921999 16169 tablet_bootstrap.cc:654] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:41.922781 16169 log.cc:824] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:41.922854 16171 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: No bootstrap required, opened a new log
I20260620 19:02:41.923017 16171 ts_tablet_manager.cc:1397] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:02:41.923447 16170 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Bootstrap starting.
I20260620 19:02:41.926471 16170 tablet_bootstrap.cc:654] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:41.927238 16170 log.cc:824] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:41.927306 16171 raft_consensus.cc:348] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.927603 16171 raft_consensus.cc:374] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:41.927650 16171 raft_consensus.cc:729] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d340c53d4c484b5ea16adea8ee9b3995, State: Initialized, Role: FOLLOWER
I20260620 19:02:41.928100 16171 consensus_queue.cc:260] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.928740 16135 heartbeater.cc:503] Master 127.2.208.190:43803 was elected leader, sending a full tablet report...
I20260620 19:02:41.930343 16171 ts_tablet_manager.cc:1428] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 19:02:41.932543 16170 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: No bootstrap required, opened a new log
I20260620 19:02:41.932724 16169 tablet_bootstrap.cc:492] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: No bootstrap required, opened a new log
I20260620 19:02:41.932896 16170 ts_tablet_manager.cc:1397] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:02:41.933100 16169 ts_tablet_manager.cc:1397] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260620 19:02:41.936980 16169 raft_consensus.cc:348] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.937037 16170 raft_consensus.cc:348] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.937300 16170 raft_consensus.cc:374] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:41.937361 16169 raft_consensus.cc:374] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:41.937422 16170 raft_consensus.cc:729] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0df86dcfc9b4e9daca046fd84036c39, State: Initialized, Role: FOLLOWER
I20260620 19:02:41.937466 16169 raft_consensus.cc:729] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de141ac2eb244611ace6bad80788eee6, State: Initialized, Role: FOLLOWER
I20260620 19:02:41.937971 16170 consensus_queue.cc:260] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.937976 16169 consensus_queue.cc:260] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:41.939100 16169 ts_tablet_manager.cc:1428] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:02:41.939100 16170 ts_tablet_manager.cc:1428] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:02:41.939473 16001 heartbeater.cc:503] Master 127.2.208.190:43803 was elected leader, sending a full tablet report...
I20260620 19:02:41.939483 15867 heartbeater.cc:503] Master 127.2.208.190:43803 was elected leader, sending a full tablet report...
I20260620 19:02:42.023485 16175 raft_consensus.cc:482] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:42.023792 16175 raft_consensus.cc:504] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:42.025195 16175 leader_election.cc:288] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de141ac2eb244611ace6bad80788eee6 (127.2.208.129:39889), a0df86dcfc9b4e9daca046fd84036c39 (127.2.208.130:34715)
I20260620 19:02:42.033823 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.026474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:44133 (local address 127.2.208.130:34715)
0620 19:02:42.026572 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:42.026579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:42.029166 (+ 2587us) server_negotiation.cc:407] Connection header received
0620 19:02:42.029350 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:42.029353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:42.029403 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:42.029450 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:42.029840 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.030531 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.032030 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.032644 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.032756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:42.032812 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:42.033572 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 19:02:42.033668 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:02:42.034351 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.025836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:34715 (local address 127.2.208.131:44133)
0620 19:02:42.029088 (+ 3252us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:42.029102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:42.029229 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:42.029581 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:42.029598 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:42.029614 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:42.029713 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:42.029720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.030667 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.030670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:42.031889 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:42.031901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.033332 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.033336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:42.033432 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:42.033455 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:02:42.034115 (+ 660us) client_negotiation.cc:240] Negotiation successful
0620 19:02:42.034189 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3189,"thread_start_us":106,"threads_started":1}
I20260620 19:02:42.034430 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.026329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:33079 (local address 127.2.208.129:39889)
0620 19:02:42.026446 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:42.026451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:42.028833 (+ 2382us) server_negotiation.cc:407] Connection header received
0620 19:02:42.028990 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:42.028994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:42.029051 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:42.029149 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:42.029558 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.030373 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.032973 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.033585 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.033666 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:42.033707 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:42.034231 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 19:02:42.034308 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:02:42.034992 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.025389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39889 (local address 127.2.208.131:33079)
0620 19:02:42.028738 (+ 3349us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:42.028753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:42.028884 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:42.029269 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:42.029273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:42.029288 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:42.029436 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:42.029445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.030500 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.030503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:42.032282 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:42.032291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.033809 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.033812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:42.033927 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:42.033947 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:02:42.034794 (+ 847us) client_negotiation.cc:240] Negotiation successful
0620 19:02:42.034872 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3276,"thread_start_us":131,"threads_started":1}
I20260620 19:02:42.035099 15955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aeb2d8e5ae42c9a924a17ae9b905c4" candidate_uuid: "d340c53d4c484b5ea16adea8ee9b3995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0df86dcfc9b4e9daca046fd84036c39" is_pre_election: true
I20260620 19:02:42.035462 15955 raft_consensus.cc:2454] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d340c53d4c484b5ea16adea8ee9b3995 in term 0.
I20260620 19:02:42.036042 16024 leader_election.cc:302] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0df86dcfc9b4e9daca046fd84036c39, d340c53d4c484b5ea16adea8ee9b3995; no voters:
I20260620 19:02:42.036372 16175 raft_consensus.cc:2781] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:02:42.036456 16175 raft_consensus.cc:482] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:02:42.036502 16175 raft_consensus.cc:3037] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:42.037904 16175 raft_consensus.cc:504] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:42.038303 16175 leader_election.cc:288] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [CANDIDATE]: Term 1 election: Requested vote from peers de141ac2eb244611ace6bad80788eee6 (127.2.208.129:39889), a0df86dcfc9b4e9daca046fd84036c39 (127.2.208.130:34715)
I20260620 19:02:42.038698 15821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aeb2d8e5ae42c9a924a17ae9b905c4" candidate_uuid: "d340c53d4c484b5ea16adea8ee9b3995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de141ac2eb244611ace6bad80788eee6" is_pre_election: true
I20260620 19:02:42.039002 15821 raft_consensus.cc:2454] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d340c53d4c484b5ea16adea8ee9b3995 in term 0.
I20260620 19:02:42.040431 15821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aeb2d8e5ae42c9a924a17ae9b905c4" candidate_uuid: "d340c53d4c484b5ea16adea8ee9b3995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de141ac2eb244611ace6bad80788eee6"
I20260620 19:02:42.040551 15821 raft_consensus.cc:3037] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:42.041893 15821 raft_consensus.cc:2454] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d340c53d4c484b5ea16adea8ee9b3995 in term 1.
I20260620 19:02:42.042269 16022 leader_election.cc:302] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d340c53d4c484b5ea16adea8ee9b3995, de141ac2eb244611ace6bad80788eee6; no voters:
I20260620 19:02:42.042572 15955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aeb2d8e5ae42c9a924a17ae9b905c4" candidate_uuid: "d340c53d4c484b5ea16adea8ee9b3995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0df86dcfc9b4e9daca046fd84036c39"
I20260620 19:02:42.042714 15955 raft_consensus.cc:3037] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:42.044090 15955 raft_consensus.cc:2454] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d340c53d4c484b5ea16adea8ee9b3995 in term 1.
I20260620 19:02:42.044579 16175 raft_consensus.cc:2781] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:42.044837 16175 raft_consensus.cc:686] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 1 LEADER]: Becoming Leader. State: Replica: d340c53d4c484b5ea16adea8ee9b3995, State: Running, Role: LEADER
I20260620 19:02:42.045205 16175 consensus_queue.cc:237] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } }
I20260620 19:02:42.049882 15668 catalog_manager.cc:5495] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 reported cstate change: term changed from 0 to 1, leader changed from <none> to d340c53d4c484b5ea16adea8ee9b3995 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "d340c53d4c484b5ea16adea8ee9b3995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d340c53d4c484b5ea16adea8ee9b3995" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 33799 } health_report { overall_health: HEALTHY } } }
W20260620 19:02:42.069365 16136 tablet.cc:2369] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:02:42.084985 15868 tablet.cc:2369] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:02:42.085748 16002 tablet.cc:2369] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:02:42.106676 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.093981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.2.208.131:33799)
0620 19:02:42.094123 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:42.094127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:42.101291 (+ 7164us) server_negotiation.cc:407] Connection header received
0620 19:02:42.101511 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:42.101515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:42.101601 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:42.101729 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:02:42.102229 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.103043 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.104426 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:42.104763 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:42.104872 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:42.105299 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:02:42.105868 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:02:42.105989 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:42.106431 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 19:02:42.106498 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:02:42.128412 15955 raft_consensus.cc:1261] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 1 FOLLOWER]: Refusing update from remote peer d340c53d4c484b5ea16adea8ee9b3995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:42.129384 15821 raft_consensus.cc:1261] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P de141ac2eb244611ace6bad80788eee6 [term 1 FOLLOWER]: Refusing update from remote peer d340c53d4c484b5ea16adea8ee9b3995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:42.130257 16175 consensus_queue.cc:1035] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de141ac2eb244611ace6bad80788eee6" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:42.131275 16175 consensus_queue.cc:1035] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0df86dcfc9b4e9daca046fd84036c39" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:42.156024 16185 mvcc.cc:205] Tried to move back new op lower bound from 7298998936065916928 to 7298998935745339392. Current Snapshot: MvccSnapshot[applied={T|T < 7298998936065916928}]
I20260620 19:02:42.167398 15669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:42.167623 15669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:42.171355 15669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99467dcc34ef4297a06040763f107081: Sending DeleteTablet for 3 replicas of tablet e0aeb2d8e5ae42c9a924a17ae9b905c4
I20260620 19:02:42.173933 15935 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:42 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:40962
I20260620 19:02:42.174424 16193 tablet_replica.cc:330] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: stopping tablet replica
I20260620 19:02:42.174780 16056 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:42 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:39042
I20260620 19:02:42.175606 15791 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:42 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:40346
I20260620 19:02:42.177700 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15707
I20260620 19:02:42.179685 16193 raft_consensus.cc:2229] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:02:42.179970 16193 raft_consensus.cc:2258] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:42.189357 16194 tablet_replica.cc:330] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: stopping tablet replica
I20260620 19:02:42.189836 16194 raft_consensus.cc:2229] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:02:42.190049 16193 ts_tablet_manager.cc:1905] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:42.190274 16194 raft_consensus.cc:2258] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:42.191833 16194 ts_tablet_manager.cc:1905] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:42.192955 16193 ts_tablet_manager.cc:1918] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:42.193109 16193 log.cc:1196] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/e0aeb2d8e5ae42c9a924a17ae9b905c4
I20260620 19:02:42.193519 16193 ts_tablet_manager.cc:1939] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P a0df86dcfc9b4e9daca046fd84036c39: Deleting consensus metadata
I20260620 19:02:42.194654 16194 ts_tablet_manager.cc:1918] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:42.194883 15655 catalog_manager.cc:4841] TS a0df86dcfc9b4e9daca046fd84036c39 (127.2.208.130:34715): tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 (table test-table [id=5e3cce7b8b1b49fda24d5cb4c43eb989]) successfully deleted
W20260620 19:02:42.195147 15653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:39889 (error 108)
I20260620 19:02:42.195988 16194 log.cc:1196] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/e0aeb2d8e5ae42c9a924a17ae9b905c4
I20260620 19:02:42.196245 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15871
I20260620 19:02:42.199493 16194 ts_tablet_manager.cc:1939] T e0aeb2d8e5ae42c9a924a17ae9b905c4 P d340c53d4c484b5ea16adea8ee9b3995: Deleting consensus metadata
I20260620 19:02:42.200850 15655 catalog_manager.cc:4841] TS d340c53d4c484b5ea16adea8ee9b3995 (127.2.208.131:33799): tablet e0aeb2d8e5ae42c9a924a17ae9b905c4 (table test-table [id=5e3cce7b8b1b49fda24d5cb4c43eb989]) successfully deleted
I20260620 19:02:42.201949 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:42.201623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39889 (local address 127.0.0.1:40358)
0620 19:02:42.201796 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:42.201864 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 19:02:42.202207 15653 catalog_manager.cc:4584] TS de141ac2eb244611ace6bad80788eee6 (127.2.208.129:39889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0aeb2d8e5ae42c9a924a17ae9b905c4: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39889: connect: Connection refused (error 111)
I20260620 19:02:42.223656 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16004
I20260620 19:02:42.230806 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 15636
I20260620 19:02:42.238838 2882 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:02:42.583343 2882 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:02:42.583491 2882 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:02:42.928483 2882 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:02:42.928 UTC [14703] LOG: received smart shutdown request
2026-06-20 19:02:42.932 UTC [14703] LOG: background worker "logical replication launcher" (PID 14712) exited with exit code 1
2026-06-20 19:02:42.932 UTC [14707] LOG: shutting down
2026-06-20 19:02:43.097 UTC [14703] LOG: database system is shut down
2026-06-20T19:02:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45952 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:02:43 dist-test-slave-tsm4 krb5kdc[16210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:02:43 dist-test-slave-tsm4 krb5kdc[16210](info): set up 2 sockets
Jun 20 19:02:43 dist-test-slave-tsm4 krb5kdc[16210](info): commencing operation
krb5kdc: starting...
W20260620 19:02:44.247550 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:02:44.365990 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:46561
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:44.470546 16223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:44.470814 16223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:44.470885 16223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:44.474841 16223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:44.474929 16223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:44.474968 16223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:44.475032 16223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:44.475068 16223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:44.480180 16223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:44.481178 16223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:44.482149 16223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:44.490445 16228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.490549 16231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.490473 16229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:44.490674 16223 server_base.cc:1034] running on GCE node
I20260620 19:02:44.491258 16223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:44.492316 16223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:44.493474 16223 hybrid_clock.cc:648] HybridClock initialized: now 1781982164493437 us; error 47 us; skew 500 ppm
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:44.496186 16223 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:44.497048 16223 webserver.cc:460] Webserver started at http://127.2.208.190:36965/ using document root <none> and password file <none>
I20260620 19:02:44.497596 16223 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:44.497684 16223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:44.497900 16223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:44.499939 16223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "9afc91ce9c1eb6110353e3ac75dd11f9"
server_key_iv: "d5c6fc0abd90a18885a33278fd15d53f"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.500478 16223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "9afc91ce9c1eb6110353e3ac75dd11f9"
server_key_iv: "d5c6fc0abd90a18885a33278fd15d53f"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.504163 16223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:02:44.506390 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:44.507552 16223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:44.507681 16223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "9afc91ce9c1eb6110353e3ac75dd11f9"
server_key_iv: "d5c6fc0abd90a18885a33278fd15d53f"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.507782 16223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:44.520532 16223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:44.523023 16223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:44.523203 16223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:44.529763 16223 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:46561
I20260620 19:02:44.529788 16290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:46561 every 8 connection(s)
I20260620 19:02:44.530617 16223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:44.531209 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16223
I20260620 19:02:44.531337 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:02:44.531672 2882 external_mini_cluster.cc:1428] Setting key b0d6bbe4b6349c3b2979c9865ff73bd3
I20260620 19:02:44.533866 16291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:44.537514 16291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.189@KRBTEST.COM" created.
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:44.552379 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.539623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46561 (local address 127.0.0.1:52192)
0620 19:02:44.540181 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.540225 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.540977 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.541674 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.541679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.542080 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.542316 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.542328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.543412 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.543416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.544566 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.544581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.545318 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.545325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.545472 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.546056 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.546080 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.547718 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.549563 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.549572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.549581 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.549789 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.550115 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.550118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.550120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.550230 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.550774 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.550780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.550920 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.551075 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.551444 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":123,"thread_start_us":133,"threads_started":1}
I20260620 19:02:44.552383 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.539623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.2.208.190:46561)
0620 19:02:44.540052 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.540066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.540325 (+ 259us) server_negotiation.cc:407] Connection header received
0620 19:02:44.541271 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.541278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.541493 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.541680 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.542455 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.543277 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.544699 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.545154 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.545276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.547866 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.547871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.547880 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.547899 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.549441 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.549898 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.549905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.549907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.550000 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.550327 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.550330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.550331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.550660 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.550799 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.551146 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.551393 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":294,"thread_start_us":166,"threads_started":1}
I20260620 19:02:44.554575 16291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.555269 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.555000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60062)
0620 19:02:44.555134 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.555213 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 19:02:44.555546 16242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
I20260620 19:02:44.556326 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.556100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60078)
0620 19:02:44.556229 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.556278 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:02:44.556468 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:46507: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
I20260620 19:02:44.557916 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:46507
--webserver_interface=127.2.208.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:44.605721 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } attempt: 1
I20260620 19:02:44.606660 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.606339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60086)
0620 19:02:44.606519 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.606583 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 19:02:44.607440 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.607150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60092)
0620 19:02:44.607255 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.607305 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:02:44.607575 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:46507: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
W20260620 19:02:44.664012 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:44.664269 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:44.664335 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:44.668283 16298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:44.668359 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:44.668402 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:44.668459 16298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:44.668498 16298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:44.673501 16298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.208.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:44.674484 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:44.675495 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:02:44.675725 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } attempt: 2
I20260620 19:02:44.676553 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.676245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60102)
0620 19:02:44.676398 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.676479 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 19:02:44.677230 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.677028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60110)
0620 19:02:44.677129 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.677182 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:02:44.677356 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:46507: Network error: Client connection negotiation failed: client connection to 127.2.208.189:46507: connect: Connection refused (error 111)
W20260620 19:02:44.683746 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.683753 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.683761 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:44.684011 16298 server_base.cc:1034] running on GCE node
I20260620 19:02:44.684576 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:44.685577 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:44.686766 16298 hybrid_clock.cc:648] HybridClock initialized: now 1781982164686741 us; error 41 us; skew 500 ppm
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:44.689491 16298 init.cc:377] Logged in from keytab as kudu/127.2.208.189@KRBTEST.COM (short username kudu)
I20260620 19:02:44.690299 16298 webserver.cc:460] Webserver started at http://127.2.208.189:43763/ using document root <none> and password file <none>
I20260620 19:02:44.690812 16298 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:44.690899 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:44.691114 16298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:44.693044 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data/instance:
uuid: "e33a1e9b86284bfa912159343362dda6"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "d14387e15e89441fb137c5cf95a88a07"
server_key_iv: "a748dbb4d1b85e031b03f465040b50fe"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.693619 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal/instance:
uuid: "e33a1e9b86284bfa912159343362dda6"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "d14387e15e89441fb137c5cf95a88a07"
server_key_iv: "a748dbb4d1b85e031b03f465040b50fe"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.697232 16298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:02:44.699558 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:44.700649 16298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:44.700783 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "e33a1e9b86284bfa912159343362dda6"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "d14387e15e89441fb137c5cf95a88a07"
server_key_iv: "a748dbb4d1b85e031b03f465040b50fe"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.700886 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:44.730041 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:44.732734 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:44.732926 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:44.739656 16298 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.189:46507
I20260620 19:02:44.739681 16365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.189:46507 every 8 connection(s)
I20260620 19:02:44.740499 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
I20260620 19:02:44.743075 16366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:44.743393 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16298
I20260620 19:02:44.743511 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-1/wal/instance
I20260620 19:02:44.743819 2882 external_mini_cluster.cc:1428] Setting key fb69adcb74a36e359b1defe5bf82a02d
I20260620 19:02:44.747306 16366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.188@KRBTEST.COM" created.
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:44.761386 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.749237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.2.208.190:46561)
0620 19:02:44.749394 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.749399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.750239 (+ 840us) server_negotiation.cc:407] Connection header received
0620 19:02:44.751478 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.751481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.751523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.751598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.752831 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.753325 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.754729 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.755150 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.755272 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.757534 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.757541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.757556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.757589 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.759194 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.759848 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.759850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.759851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.759918 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.760308 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.760310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.760311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.760462 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.760528 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.761173 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.761267 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:02:44.762457 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.749570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46561 (local address 127.0.0.1:52194)
0620 19:02:44.750063 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.750120 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.751245 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.751805 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.751816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.752193 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.752707 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.752721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.753439 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.753441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.754613 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.754622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.755288 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.755291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.755485 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.756042 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.756057 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.757374 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.759311 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.759320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.759336 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.759738 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.760019 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.760024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.760029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.760215 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.760564 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.760573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.760866 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.761077 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.761414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260620 19:02:44.763803 16366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.768522 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } attempt: 3
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
I20260620 19:02:44.771126 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:35607
--webserver_interface=127.2.208.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:44.773227 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.764414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60122)
0620 19:02:44.764539 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.764556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.764647 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.765407 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.765413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.765564 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.765644 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.765649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.766634 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.766636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.767150 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.767158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.767884 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.767887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.767955 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.768317 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.768327 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.769665 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.771701 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.771707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.771709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.771888 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.772197 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.772203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.772207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.772310 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.772764 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.772766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.772842 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.772960 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.773072 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:02:44.773366 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.764726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.2.208.189:46507)
0620 19:02:44.765060 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.765075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.765093 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:02:44.765124 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.765128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.765262 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.765407 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.765749 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.766535 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.767249 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.767788 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.767872 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.769782 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.769790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.769801 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.769822 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.771601 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.771989 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.771994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.771995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.772098 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.772420 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.772424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.772427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.772663 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.772797 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.773042 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.773157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:02:44.775444 16366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.776464 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.775904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38304)
0620 19:02:44.776283 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.776368 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 19:02:44.776902 16317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
I20260620 19:02:44.777878 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.777582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38314)
0620 19:02:44.777719 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.777801 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:02:44.778126 16366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
I20260620 19:02:44.779182 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.768970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60134)
0620 19:02:44.769062 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.769076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.769192 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.769635 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.769640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.769785 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.770377 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.770387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.772083 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.772087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.772964 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.772977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.773645 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.773649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.773740 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.774153 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.774166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.775622 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.777598 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.777602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.777605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.777842 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.778216 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.778219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.778222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.778313 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.778693 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.778696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.778795 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.778897 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.779022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:02:44.779206 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.769033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.2.208.189:46507)
0620 19:02:44.769414 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.769422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.769436 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:02:44.769464 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.769467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.769507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.769589 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.770494 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.771908 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.773120 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.773513 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.773956 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.775771 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.775774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.775776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.775795 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.777475 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.777979 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.777983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.777986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.778076 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.778434 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.778439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.778440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.778597 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.778701 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.778957 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.779069 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":176,"threads_started":1}
I20260620 19:02:44.779877 16291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.780568 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.780285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38326)
0620 19:02:44.780413 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.780476 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:02:44.781260 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.780973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38334)
0620 19:02:44.781129 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.781186 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 19:02:44.781387 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
I20260620 19:02:44.812557 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 1
I20260620 19:02:44.813280 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.812994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38340)
0620 19:02:44.813138 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.813199 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 19:02:44.814000 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.813722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38348)
0620 19:02:44.813862 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.813923 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 19:02:44.814142 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
I20260620 19:02:44.827414 16366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 1
I20260620 19:02:44.828097 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.827842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38364)
0620 19:02:44.827972 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.828028 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:02:44.828730 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.828531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38376)
0620 19:02:44.828629 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.828676 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:02:44.828862 16366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
W20260620 19:02:44.878428 16373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:44.878655 16373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:44.878723 16373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:44.882849 16373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:44.882902 16373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:44.882923 16373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:44.882982 16373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:44.883011 16373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:44.888180 16373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46561,127.2.208.189:46507,127.2.208.188:35607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.208.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:44.889089 16373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:44.889300 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 2
I20260620 19:02:44.890143 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.889815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38382)
0620 19:02:44.889959 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.890041 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 19:02:44.890357 16373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:02:44.891491 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.890996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38388)
0620 19:02:44.891343 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.891409 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
W20260620 19:02:44.891686 16291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
I20260620 19:02:44.897063 16366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 2
I20260620 19:02:44.897835 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.897505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38392)
0620 19:02:44.897724 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.897775 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 19:02:44.898520 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.898325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38398)
0620 19:02:44.898420 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.898463 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:02:44.898682 16366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35607: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35607: connect: Connection refused (error 111)
W20260620 19:02:44.900187 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.900166 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:44.900193 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:44.900859 16373 server_base.cc:1034] running on GCE node
I20260620 19:02:44.901502 16373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:44.902688 16373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:44.903860 16373 hybrid_clock.cc:648] HybridClock initialized: now 1781982164903833 us; error 43 us; skew 500 ppm
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:44.906606 16373 init.cc:377] Logged in from keytab as kudu/127.2.208.188@KRBTEST.COM (short username kudu)
I20260620 19:02:44.907495 16373 webserver.cc:460] Webserver started at http://127.2.208.188:35111/ using document root <none> and password file <none>
I20260620 19:02:44.907981 16373 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:44.908035 16373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:44.908200 16373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:44.910027 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data/instance:
uuid: "6c8552e60ef74abeb89d708b7a8c2ad1"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "1c42fd86f9b1a6dc79a5ab032a484b61"
server_key_iv: "0c6116692a0cc67c573542d3da873a97"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.910560 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal/instance:
uuid: "6c8552e60ef74abeb89d708b7a8c2ad1"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "1c42fd86f9b1a6dc79a5ab032a484b61"
server_key_iv: "0c6116692a0cc67c573542d3da873a97"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.913906 16373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 19:02:44.916055 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:44.917124 16373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:44.917248 16373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "6c8552e60ef74abeb89d708b7a8c2ad1"
format_stamp: "Formatted at 2026-06-20 19:02:44 on dist-test-slave-tsm4"
server_key: "1c42fd86f9b1a6dc79a5ab032a484b61"
server_key_iv: "0c6116692a0cc67c573542d3da873a97"
server_key_version: "encryptionkey@0"
I20260620 19:02:44.917347 16373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:44.927676 16373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:44.930657 16373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:44.930847 16373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:44.937595 16373 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.188:35607
I20260620 19:02:44.937610 16441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.188:35607 every 8 connection(s)
I20260620 19:02:44.938871 16373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
I20260620 19:02:44.941432 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:44.944885 16442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.948442 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16373
I20260620 19:02:44.948577 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982008625501-2882-0/minicluster-data/master-2/wal/instance
I20260620 19:02:44.948866 2882 external_mini_cluster.cc:1428] Setting key 3668d7acd39b8cf6538f81290062614b
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:02:44.958001 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.946648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.2.208.190:46561)
0620 19:02:44.946787 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.946791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.947564 (+ 773us) server_negotiation.cc:407] Connection header received
0620 19:02:44.948561 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.948565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.948651 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.948758 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.950091 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.950594 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.951751 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.951965 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.952077 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.954427 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.954430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.954432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.954452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.955955 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.956434 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.956440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.956443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.956550 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.956913 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.956934 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.956938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.957198 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.957308 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.957754 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.957874 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:02:44.958914 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.946917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46561 (local address 127.0.0.1:52198)
0620 19:02:44.947340 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.947453 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.948291 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.948922 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.948930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.949320 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.949940 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.949961 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.950714 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.950717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.951635 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.951644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.952107 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.952110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.952273 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.952838 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.952855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.954284 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.956060 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.956070 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.956078 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.956303 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.956654 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.956657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.956658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.956811 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.957310 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.957315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.957517 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.957666 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.957945 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260620 19:02:44.959642 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.949495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.2.208.188:35607)
0620 19:02:44.949810 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.949820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.949839 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:02:44.949951 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.949955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.950095 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.950248 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.950904 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.951756 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.952598 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.953073 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.953168 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.955641 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.955647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.955660 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.955687 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.957747 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.958186 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.958192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.958193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.958270 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.958528 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.958532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.958533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.958774 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.958944 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.959315 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.959474 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":235,"thread_start_us":128,"threads_started":1}
I20260620 19:02:44.960289 16442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:44.968473 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.960745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:46507 (local address 127.0.0.1:60138)
0620 19:02:44.960872 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.960882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.960964 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.961212 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.961215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.961370 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.961502 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.961512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.962395 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.962399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.962932 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.962941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.963421 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.963424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.963526 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.963890 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.963903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.965144 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.966999 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.967004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.967007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.967227 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.967563 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.967567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.967570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.967672 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.968084 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.968086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.968157 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.968232 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.968318 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 19:02:44.968550 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.960853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.2.208.189:46507)
0620 19:02:44.960985 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.960991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.961007 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:02:44.961051 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.961053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.961102 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.961201 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.961627 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.962288 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.963072 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.963292 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.963408 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.965284 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.965288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.965293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.965328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.966899 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.967326 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.967332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.967337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.967464 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.967766 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.967769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.967771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.967978 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.968087 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.968308 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.968430 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:02:44.969511 16442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:02:44.969975 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.961665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.2.208.190:46561)
0620 19:02:44.961761 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.961765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.961777 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:44.961866 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.961869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.961905 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.961979 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.962516 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.962935 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.963884 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.964080 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.964153 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.966599 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.966606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.966611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.966638 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.968319 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.968781 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.968784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.968785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.968860 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.969144 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.969146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.969147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.969361 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.969564 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.969691 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.969809 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:02:44.977981 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.969960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38416)
0620 19:02:44.970049 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.970061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.970126 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.970638 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.970640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.970813 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.970917 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.970923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.971947 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.971949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.972718 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.972727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.973291 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.973293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.973390 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.973776 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:44.973789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:44.974882 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:44.976605 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.976607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.976609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.976834 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.977147 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:44.977149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:44.977150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:44.977223 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:44.977647 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:44.977649 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:44.977730 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 19:02:44.977808 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:02:44.977888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":18}
I20260620 19:02:44.978163 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.970018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.2.208.188:35607)
0620 19:02:44.970110 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.970114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.970125 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:02:44.970202 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.970205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.970240 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.970323 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.971041 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.971842 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.972845 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.973188 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.973286 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.975019 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.975024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.975029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.975057 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.976490 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.976948 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.976956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.976957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.977047 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.977316 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.977319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.977321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.977533 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.977649 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.977889 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.977998 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":38}
I20260620 19:02:44.979674 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.971059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.2.208.189:46507)
0620 19:02:44.971164 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.971168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.971180 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:44.971283 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.971285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.971315 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.971431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.972078 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.972781 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.973704 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.974018 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.974121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.976566 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.976570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.976572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.976594 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.978186 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.978576 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.978579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.978581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.978657 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.978911 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.978915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.978917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.979073 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.979255 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.979402 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.979504 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:02:44.981238 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: Bootstrap starting.
I20260620 19:02:44.982527 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:02:44.983413 16442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:44.984135 16442 log.cc:824] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:44.986901 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: No bootstrap required, opened a new log
I20260620 19:02:44.989858 16366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 3
I20260620 19:02:44.990949 16291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } attempt: 3
I20260620 19:02:44.991400 16442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:44.991698 16442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:44.991770 16442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8552e60ef74abeb89d708b7a8c2ad1, State: Initialized, Role: FOLLOWER
I20260620 19:02:44.992434 16442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:44.993278 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.984838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.2.208.190:46561)
0620 19:02:44.984937 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.984941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.984953 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:44.984985 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.984989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.985027 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.985122 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.985891 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.986558 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.987666 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.987957 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.988028 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.989724 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.989728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.989731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.989755 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.991400 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.991959 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.991962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.991964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.992062 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.992364 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.992367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.992369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.992569 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.992672 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.993010 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.993130 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:02:44.993805 16442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:44.994832 16455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:44.994961 16455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:44.995216 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.984574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.2.208.189:46507)
0620 19:02:44.984763 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.984769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.984839 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:02:44.985012 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.985016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.985057 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.985113 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.986017 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.986762 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.987853 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.988169 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.988247 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.989930 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.989933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.989936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.989966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.991672 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.993795 (+ 2123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.993799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.993802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.993895 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.994210 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.994215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.994220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.994450 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.994567 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.994948 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.995072 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 19:02:44.997408 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.984898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.2.208.188:35607)
0620 19:02:44.984991 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.984994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.985375 (+ 381us) server_negotiation.cc:407] Connection header received
0620 19:02:44.985547 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.985552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.985598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.985648 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.986502 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.987185 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.988297 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.989158 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.989246 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:44.992487 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:44.992491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:44.992494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:44.992517 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:44.994525 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.996047 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.996052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.996056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.996141 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:44.996453 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:44.996456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:44.996476 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:44.996723 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:44.996829 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:44.997171 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 19:02:44.997267 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:02:44.997442 16373 master_runner.cc:186] Error getting master registration for 127.2.208.190:46561: OK, instance_id { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" instance_seqno: 1781982164527882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260620 19:02:45.000272 16468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:02:45.000352 16468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:02:45.000419 16373 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
Jun 20 19:02:44 dist-test-slave-tsm4 krb5kdc[16210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:02:45.007053 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.991977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38434)
0620 19:02:44.992524 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.992537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.992608 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.996475 (+ 3867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.996478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.996655 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.996752 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.996760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.998870 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.998874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:44.999656 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.999665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:45.000232 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:45.000236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:45.000319 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:45.000874 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:45.000888 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:45.002293 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:45.005334 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:45.005338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:45.005341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:45.005586 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:45.005983 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:45.005986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:45.005988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:45.006082 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:45.006584 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:45.006587 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:45.006676 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:02:45.006737 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:02:45.006874 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461}
I20260620 19:02:45.007079 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.992246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35607 (local address 127.0.0.1:38450)
0620 19:02:44.992763 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:44.992776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:44.992842 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:44.995460 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:44.995463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:44.995629 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:44.995731 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:44.995740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.999496 (+ 3756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.999499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:45.000266 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:45.000275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:45.000890 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:45.000893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:45.000972 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:45.001396 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:45.001410 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:45.002921 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:45.005507 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:45.005511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:45.005513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:45.005763 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:45.006062 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:45.006065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:45.006068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:45.006158 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:45.006601 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:45.006604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:45.006692 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:02:45.006797 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:02:45.006915 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438}
I20260620 19:02:45.007615 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.993204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.2.208.188:35607)
0620 19:02:44.995205 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.995212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.995224 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:44.995255 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.995258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.995297 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.995841 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.995959 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.999350 (+ 3391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:45.000391 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:45.000768 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:45.000876 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:45.003062 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:45.003068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:45.003073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:45.003323 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:45.005401 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:45.005879 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:45.005883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:45.005885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:45.005976 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:45.006267 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:45.006270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:45.006271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:45.006470 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:45.006566 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:45.006868 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 19:02:45.006963 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":373,"server-negotiator.queue_time_us":1944,"spinlock_wait_cycles":47872,"thread_start_us":93,"threads_started":1}
I20260620 19:02:45.007959 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:44.992995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.2.208.188:35607)
0620 19:02:44.994920 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:44.994933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:44.994945 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:44.994974 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:44.994977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:44.995015 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:44.996857 (+ 1842us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:44.996959 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:44.998768 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:44.999773 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:45.000118 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:45.002709 (+ 2591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:45.002826 (+ 117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:45.002830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:45.002832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:45.002853 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:45.005203 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:45.005712 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:45.005715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:45.005716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:45.005830 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:45.006192 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:45.006194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:45.006195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:45.006467 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:45.007605 (+ 1138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:45.007677 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:02:45.007780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1857,"thread_start_us":93,"threads_started":1}
I20260620 19:02:45.011030 16291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: Bootstrap starting.
I20260620 19:02:45.011031 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: Bootstrap starting.
I20260620 19:02:45.013715 16366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:45.013715 16291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:45.014824 16291 log.cc:824] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:45.016774 16366 log.cc:824] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:45.017659 16291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: No bootstrap required, opened a new log
I20260620 19:02:45.021165 16291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.021390 16291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:45.021446 16291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eb07cd13e44ec6a39cd5ccd2c8dcab, State: Initialized, Role: FOLLOWER
I20260620 19:02:45.021924 16291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.022393 16472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.022527 16472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.023114 16291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:45.025301 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: No bootstrap required, opened a new log
W20260620 19:02:45.027815 16483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:02:45.027890 16483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:02:45.029139 16366 raft_consensus.cc:348] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.029350 16366 raft_consensus.cc:374] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260620 19:02:45.029543 16244 tablet.cc:2369] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:02:45.029990 16366 raft_consensus.cc:729] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33a1e9b86284bfa912159343362dda6, State: Initialized, Role: FOLLOWER
I20260620 19:02:45.032104 16366 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.033362 16366 sys_catalog.cc:564] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:45.032295 16223 master_runner.cc:186] Error getting master registration for 127.2.208.189:46507: OK, instance_id { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" instance_seqno: 1781982164737801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:02:45.035466 16223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:02:45.038738 16496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:02:45.038810 16496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:02:45.040058 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.040169 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.041157 16298 master_runner.cc:263] No leader master found from master e33a1e9b86284bfa912159343362dda6
I20260620 19:02:45.041213 16298 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:02:45.145270 16472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:45.145509 16472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.146652 16472 leader_election.cc:288] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e33a1e9b86284bfa912159343362dda6 (127.2.208.189:46507), 6c8552e60ef74abeb89d708b7a8c2ad1 (127.2.208.188:35607)
I20260620 19:02:45.147514 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33a1e9b86284bfa912159343362dda6" is_pre_election: true
I20260620 19:02:45.147855 16340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12eb07cd13e44ec6a39cd5ccd2c8dcab in term 0.
I20260620 19:02:45.148753 16416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" is_pre_election: true
I20260620 19:02:45.149039 16416 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12eb07cd13e44ec6a39cd5ccd2c8dcab in term 0.
I20260620 19:02:45.149587 16242 leader_election.cc:302] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12eb07cd13e44ec6a39cd5ccd2c8dcab, 6c8552e60ef74abeb89d708b7a8c2ad1; no voters:
I20260620 19:02:45.150086 16472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:02:45.150178 16472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:02:45.150229 16472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:45.153945 16472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.154331 16472 leader_election.cc:288] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [CANDIDATE]: Term 1 election: Requested vote from peers e33a1e9b86284bfa912159343362dda6 (127.2.208.189:46507), 6c8552e60ef74abeb89d708b7a8c2ad1 (127.2.208.188:35607)
I20260620 19:02:45.156217 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33a1e9b86284bfa912159343362dda6"
I20260620 19:02:45.156343 16340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:45.157529 16340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12eb07cd13e44ec6a39cd5ccd2c8dcab in term 1.
I20260620 19:02:45.154923 16416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1"
I20260620 19:02:45.157827 16416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:45.158949 16416 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12eb07cd13e44ec6a39cd5ccd2c8dcab in term 1.
I20260620 19:02:45.159233 16242 leader_election.cc:302] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12eb07cd13e44ec6a39cd5ccd2c8dcab, 6c8552e60ef74abeb89d708b7a8c2ad1; no voters:
I20260620 19:02:45.160403 16472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:45.160632 16472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [term 1 LEADER]: Becoming Leader. State: Replica: 12eb07cd13e44ec6a39cd5ccd2c8dcab, State: Running, Role: LEADER
I20260620 19:02:45.160946 16472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } }
I20260620 19:02:45.162501 16472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb07cd13e44ec6a39cd5ccd2c8dcab. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.162601 16472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:45.162930 16500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:45.163892 16500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:45.171101 16416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [term 1 FOLLOWER]: Refusing update from remote peer 12eb07cd13e44ec6a39cd5ccd2c8dcab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:45.171744 16472 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:45.176172 16455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb07cd13e44ec6a39cd5ccd2c8dcab. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.176296 16455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.177246 16340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [term 1 FOLLOWER]: Refusing update from remote peer 12eb07cd13e44ec6a39cd5ccd2c8dcab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:45.180083 16472 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:02:45.182122 16395 tablet.cc:2369] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:02:45.185575 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb07cd13e44ec6a39cd5ccd2c8dcab. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.185707 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.190052 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.190176 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:45.191309 16455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.191447 16455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.191609 16500 catalog_manager.cc:1347] Generated new cluster ID: 2d7abcc8bd49479da4b10021b9f3e385
I20260620 19:02:45.191663 16500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:45.196245 16508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.196455 16508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:45.196944 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12eb07cd13e44ec6a39cd5ccd2c8dcab" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46561 } } peers { permanent_uuid: "e33a1e9b86284bfa912159343362dda6" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 46507 } } peers { permanent_uuid: "6c8552e60ef74abeb89d708b7a8c2ad1" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35607 } } }
I20260620 19:02:45.197057 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:02:45.197512 16504 mvcc.cc:205] Tried to move back new op lower bound from 7298998948533776384 to 7298998948501831680. Current Snapshot: MvccSnapshot[applied={T|T < 7298998948533776384}]
I20260620 19:02:45.202975 16501 mvcc.cc:205] Tried to move back new op lower bound from 7298998948533776384 to 7298998948501831680. Current Snapshot: MvccSnapshot[applied={T|T < 7298998948533776384 or (T in {7298998948533776384})}]
I20260620 19:02:45.203466 16502 mvcc.cc:205] Tried to move back new op lower bound from 7298998948533776384 to 7298998948501831680. Current Snapshot: MvccSnapshot[applied={T|T < 7298998948533776384}]
I20260620 19:02:45.214800 16500 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:02:45.215950 16500 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:45.227809 16500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12eb07cd13e44ec6a39cd5ccd2c8dcab: Generated new TSK 0
I20260620 19:02:45.228581 16500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:02:46.001691 16468 catalog_manager.cc:1259] Loaded cluster ID: 2d7abcc8bd49479da4b10021b9f3e385
I20260620 19:02:46.001780 16468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: loading cluster ID for follower catalog manager: success
I20260620 19:02:46.003464 16468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: acquiring CA information for follower catalog manager: success
I20260620 19:02:46.004850 16468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c8552e60ef74abeb89d708b7a8c2ad1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:02:46.040109 16496 catalog_manager.cc:1259] Loaded cluster ID: 2d7abcc8bd49479da4b10021b9f3e385
I20260620 19:02:46.040226 16496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: loading cluster ID for follower catalog manager: success
I20260620 19:02:46.041890 16496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: acquiring CA information for follower catalog manager: success
I20260620 19:02:46.043097 16496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e33a1e9b86284bfa912159343362dda6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:02:46.289031 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16223
I20260620 19:02:46.295882 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16298
I20260620 19:02:46.302083 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16373
2026-06-20T19:02:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3087 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:02:46 dist-test-slave-tsm4 krb5kdc[16516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:02:46 dist-test-slave-tsm4 krb5kdc[16516](info): set up 2 sockets
Jun 20 19:02:46 dist-test-slave-tsm4 krb5kdc[16516](info): commencing operation
krb5kdc: starting...
W20260620 19:02:47.327690 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:02:47.444577 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:33373
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:33373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:47.549031 16529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:47.549240 16529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:47.549278 16529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:47.553272 16529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:47.553328 16529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:47.553347 16529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:47.553393 16529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:47.553423 16529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:47.558396 16529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:33373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:33373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:47.559319 16529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:47.560266 16529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:47.568228 16535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:47.568233 16534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:47.568421 16529 server_base.cc:1034] running on GCE node
W20260620 19:02:47.568545 16537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:47.569092 16529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:47.569990 16529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:47.571139 16529 hybrid_clock.cc:648] HybridClock initialized: now 1781982167571114 us; error 38 us; skew 500 ppm
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:47.573695 16529 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:47.574458 16529 webserver.cc:460] Webserver started at http://127.2.208.190:42891/ using document root <none> and password file <none>
I20260620 19:02:47.574946 16529 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:47.575002 16529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:47.575234 16529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:47.577126 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "c2ff677b087849e28c3c0449803cc0dc"
format_stamp: "Formatted at 2026-06-20 19:02:47 on dist-test-slave-tsm4"
server_key: "03992700f84a1af9e57b03923d474308"
server_key_iv: "124b70a50771d02ede458783b7827181"
server_key_version: "encryptionkey@0"
I20260620 19:02:47.577661 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "c2ff677b087849e28c3c0449803cc0dc"
format_stamp: "Formatted at 2026-06-20 19:02:47 on dist-test-slave-tsm4"
server_key: "03992700f84a1af9e57b03923d474308"
server_key_iv: "124b70a50771d02ede458783b7827181"
server_key_version: "encryptionkey@0"
I20260620 19:02:47.581020 16529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 19:02:47.583292 16544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:47.584415 16529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:47.584563 16529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "c2ff677b087849e28c3c0449803cc0dc"
format_stamp: "Formatted at 2026-06-20 19:02:47 on dist-test-slave-tsm4"
server_key: "03992700f84a1af9e57b03923d474308"
server_key_iv: "124b70a50771d02ede458783b7827181"
server_key_version: "encryptionkey@0"
I20260620 19:02:47.584666 16529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:47.602258 16529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:47.604791 16529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:47.604979 16529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:47.611883 16529 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:33373
I20260620 19:02:47.611907 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:33373 every 8 connection(s)
I20260620 19:02:47.612676 16529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:47.615072 16597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:47.619858 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16529
I20260620 19:02:47.619983 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:02:47.620245 2882 external_mini_cluster.cc:1428] Setting key 29b30d2ad26030d3cf5129b8176d6922
I20260620 19:02:47.620558 16597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc: Bootstrap starting.
I20260620 19:02:47.623078 16597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:47.623914 16597 log.cc:824] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:47.625908 16597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc: No bootstrap required, opened a new log
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:47.630119 16597 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } }
I20260620 19:02:47.630383 16597 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:47.630442 16597 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff677b087849e28c3c0449803cc0dc, State: Initialized, Role: FOLLOWER
I20260620 19:02:47.630949 16597 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } }
I20260620 19:02:47.631162 16597 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:47.631235 16597 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:47.631336 16597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:47.632421 16597 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } }
I20260620 19:02:47.632802 16597 leader_election.cc:302] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ff677b087849e28c3c0449803cc0dc; no voters:
I20260620 19:02:47.633131 16597 leader_election.cc:288] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:02:47.633229 16602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:47.633450 16602 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [term 1 LEADER]: Becoming Leader. State: Replica: c2ff677b087849e28c3c0449803cc0dc, State: Running, Role: LEADER
I20260620 19:02:47.633785 16602 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } }
I20260620 19:02:47.634374 16597 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:47.635483 16604 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ff677b087849e28c3c0449803cc0dc. Latest consensus state: current_term: 1 leader_uuid: "c2ff677b087849e28c3c0449803cc0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } } }
I20260620 19:02:47.635625 16604 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:47.635953 16603 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2ff677b087849e28c3c0449803cc0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ff677b087849e28c3c0449803cc0dc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 33373 } } }
I20260620 19:02:47.636039 16603 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:47.636049 16611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:47.635524 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:47.621512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.2.208.190:33373)
0620 19:02:47.621890 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:47.621908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:47.621947 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:02:47.622788 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:47.622809 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:47.623192 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:47.623562 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:47.624219 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:47.625543 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:47.626437 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:47.627148 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:47.627296 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:47.629913 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:47.629918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:47.629933 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:47.629961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:47.632003 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:47.632401 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:47.632407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:47.632413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:47.632579 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:47.632827 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:47.632831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:47.632833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:47.633322 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:47.634084 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:47.634226 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:02:47.634522 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260620 19:02:47.639123 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:02:47.640252 16611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:47.645244 16611 catalog_manager.cc:1347] Generated new cluster ID: 7826f0748b174ef4a19e49cf0c8cbaca
I20260620 19:02:47.645303 16611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:47.689663 16611 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:02:47.690598 16611 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:47.699038 16611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ff677b087849e28c3c0449803cc0dc: Generated new TSK 0
I20260620 19:02:47.699731 16611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:02:47.746153 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16529
2026-06-20T19:02:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1442 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16624](info): set up 2 sockets
Jun 20 19:02:47 dist-test-slave-tsm4 krb5kdc[16624](info): commencing operation
krb5kdc: starting...
W20260620 19:02:48.771445 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:02:48 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:02:48.886052 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:48.991209 16637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:48.991480 16637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:48.991519 16637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:48.995528 16637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:48.995574 16637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:48.995594 16637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:48.995609 16637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:48.995671 16637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:49.000856 16637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:49.001767 16637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:49.002705 16637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:49.010699 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.010929 16637 server_base.cc:1034] running on GCE node
W20260620 19:02:49.010706 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.010730 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.011668 16637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:49.012732 16637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:49.013909 16637 hybrid_clock.cc:648] HybridClock initialized: now 1781982169013884 us; error 38 us; skew 500 ppm
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:49.019465 16637 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:49.020263 16637 webserver.cc:460] Webserver started at http://127.2.208.190:39179/ using document root <none> and password file <none>
I20260620 19:02:49.020741 16637 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:49.020795 16637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:49.020943 16637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:49.022823 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "993494df07cf98d4dd5984d016e939ce"
server_key_iv: "34bc972f13c66f94700a7f600cba63ed"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.023298 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "993494df07cf98d4dd5984d016e939ce"
server_key_iv: "34bc972f13c66f94700a7f600cba63ed"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.026759 16637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:02:49.029037 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.030418 16637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:02:49.030560 16637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "993494df07cf98d4dd5984d016e939ce"
server_key_iv: "34bc972f13c66f94700a7f600cba63ed"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.030675 16637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:49.052616 16637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:49.055269 16637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:49.055488 16637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:49.062218 16637 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:41125
I20260620 19:02:49.062496 16704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:41125 every 8 connection(s)
I20260620 19:02:49.063149 16637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:49.065685 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:49.070708 16705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap starting.
I20260620 19:02:49.071511 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16637
I20260620 19:02:49.071619 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:02:49.071939 2882 external_mini_cluster.cc:1428] Setting key b31ebef52de5b2fef773aefa3cc313e4
I20260620 19:02:49.073396 16705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:49.074122 16705 log.cc:824] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:49.076174 16705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: No bootstrap required, opened a new log
I20260620 19:02:49.080458 16705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:49.080639 16705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:49.080691 16705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Initialized, Role: FOLLOWER
I20260620 19:02:49.081079 16705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:49.081241 16705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:49.081329 16705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:49.081554 16705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:49.082403 16705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:49.082741 16705 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfc7c2b39ef4ac6ba03417a01d35f9e; no voters:
I20260620 19:02:49.083048 16705 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:02:49.083118 16710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:49.083345 16710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 LEADER]: Becoming Leader. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Running, Role: LEADER
I20260620 19:02:49.083714 16710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:49.084337 16705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:49.084852 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfc7c2b39ef4ac6ba03417a01d35f9e. Latest consensus state: current_term: 1 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:49.085574 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:49.085449 16711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:49.085865 16711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:49.086791 16719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:49.086869 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.072978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.2.208.190:41125)
0620 19:02:49.073362 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.073377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.073410 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:02:49.074098 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.074115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.074504 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.074869 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:49.075706 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.076687 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.077847 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.078294 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.078406 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.081269 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:49.081274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:49.081286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:49.081316 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:49.083248 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.083746 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.083751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.083757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.083888 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.084148 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.084151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.084153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.084607 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:49.084846 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.085128 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.085473 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260620 19:02:49.090503 16719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:49.095258 16719 catalog_manager.cc:1347] Generated new cluster ID: 606141f66368460394a74a57ea3ab283
I20260620 19:02:49.095314 16719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:49.113173 16719 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:02:49.114481 16719 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:49.123677 16719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Generated new TSK 0
I20260620 19:02:49.124409 16719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:02:49.162626 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:49.268157 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:49.268380 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:49.268420 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:49.272198 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:49.272240 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:49.272292 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:02:49.277766 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:49.278635 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:49.279515 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:49.288676 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.288847 16731 server_base.cc:1034] running on GCE node
W20260620 19:02:49.288684 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.288692 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.289420 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:49.290001 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:49.291154 16731 hybrid_clock.cc:648] HybridClock initialized: now 1781982169291126 us; error 41 us; skew 500 ppm
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:49.293838 16731 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:02:49.294700 16731 webserver.cc:460] Webserver started at http://127.2.208.129:44373/ using document root <none> and password file <none>
I20260620 19:02:49.295251 16731 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:49.295336 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:49.295590 16731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:49.297602 16731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "3c41b98d7c08499d9842c75a94f50129"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "05650ab1833b4c7f1b6bd19487de83c0"
server_key_iv: "c655451df76e322671d1e2da6e52907f"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.298161 16731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "3c41b98d7c08499d9842c75a94f50129"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "05650ab1833b4c7f1b6bd19487de83c0"
server_key_iv: "c655451df76e322671d1e2da6e52907f"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.301724 16731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:02:49.304029 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.305148 16731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:49.305294 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "3c41b98d7c08499d9842c75a94f50129"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "05650ab1833b4c7f1b6bd19487de83c0"
server_key_iv: "c655451df76e322671d1e2da6e52907f"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.305419 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:49.328881 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:49.331560 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:49.331748 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:49.332295 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:49.333071 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:49.333140 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.333205 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:49.333248 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.341887 16731 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:34857
I20260620 19:02:49.341924 16859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:34857 every 8 connection(s)
I20260620 19:02:49.342737 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:02:49.348476 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16731
I20260620 19:02:49.348623 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:02:49.348925 2882 external_mini_cluster.cc:1428] Setting key 2f4f209ba91166553141fbbeadf4a9ea
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:49.355875 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.344445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59715 (local address 127.2.208.190:41125)
0620 19:02:49.344607 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.344612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.345566 (+ 954us) server_negotiation.cc:407] Connection header received
0620 19:02:49.346344 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.346346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.346390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.346471 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:49.347585 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.348092 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.349583 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.349856 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.349971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.352416 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:49.352420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:49.352422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:49.352454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:49.353820 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.354334 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.354338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.354340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.354416 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.354740 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.354744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.354746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.354912 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:49.355023 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.355568 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.355698 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":65}
I20260620 19:02:49.356565 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.344822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:59715)
0620 19:02:49.345407 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.345463 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.346172 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.346615 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.346621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.346966 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:49.347452 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.347466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.348229 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.348235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.349475 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.349490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.350010 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.350014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.350224 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.350857 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:49.350872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:49.352248 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:49.353971 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.353976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.353990 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.354233 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.354507 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.354512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.354514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.354646 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.355021 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:49.355028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:49.355250 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.355493 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.355761 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:02:49.357762 16860 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
I20260620 19:02:49.358008 16860 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:49.358466 16860 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:02:49.360105 16669 ts_manager.cc:194] Registered new tserver with Master: 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857)
I20260620 19:02:49.361239 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:59715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:02:49.373756 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:49.477511 16865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:49.477725 16865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:49.477766 16865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:49.481596 16865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:49.481643 16865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:49.481698 16865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:02:49.486914 16865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:49.487813 16865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:49.488730 16865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:49.497622 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.497622 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.497594 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.497668 16865 server_base.cc:1034] running on GCE node
I20260620 19:02:49.498202 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:49.498800 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:49.499996 16865 hybrid_clock.cc:648] HybridClock initialized: now 1781982169499989 us; error 28 us; skew 500 ppm
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:49.502619 16865 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:02:49.503479 16865 webserver.cc:460] Webserver started at http://127.2.208.130:45633/ using document root <none> and password file <none>
I20260620 19:02:49.503983 16865 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:49.504066 16865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:49.504276 16865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:49.506141 16865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "412b00997cfa46ec81b6d76b036da116"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "af0323662b96cb61de01e1e5e6f5c240"
server_key_iv: "4ad1c17034a0b8266d384899294fd1c5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.506680 16865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "412b00997cfa46ec81b6d76b036da116"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "af0323662b96cb61de01e1e5e6f5c240"
server_key_iv: "4ad1c17034a0b8266d384899294fd1c5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.510315 16865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:02:49.512588 16880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.513716 16865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:02:49.513844 16865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "412b00997cfa46ec81b6d76b036da116"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "af0323662b96cb61de01e1e5e6f5c240"
server_key_iv: "4ad1c17034a0b8266d384899294fd1c5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.513944 16865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:49.529013 16865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:49.532234 16865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:49.532419 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:49.532944 16865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:49.533738 16865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:49.533807 16865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.533874 16865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:49.533926 16865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.542426 16865 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44027
I20260620 19:02:49.542445 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44027 every 8 connection(s)
I20260620 19:02:49.543205 16865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:02:49.549041 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16865
I20260620 19:02:49.549130 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:02:49.549386 2882 external_mini_cluster.cc:1428] Setting key 8529094c01bce14bf42bcbcfccdfe86a
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:49.556959 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.544930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:48523 (local address 127.2.208.190:41125)
0620 19:02:49.545102 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.545107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.546066 (+ 959us) server_negotiation.cc:407] Connection header received
0620 19:02:49.546902 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.546904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.546947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.547021 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:49.548230 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.548742 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.550074 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.550313 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.550394 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.553096 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:49.553102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:49.553106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:49.553135 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:49.554575 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.555188 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.555191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.555192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.555265 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.555731 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.555738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.555742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.555981 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:49.556066 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.556710 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.556801 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:02:49.558000 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.545277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:48523)
0620 19:02:49.545910 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.545948 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.546713 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.547176 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.547182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.547645 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:49.548100 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.548113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.548870 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.548873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.549963 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.549980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.550467 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.550470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.550671 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.551324 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:49.551341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:49.552928 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:49.554689 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.554698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.554714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.555080 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.555353 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.555357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.555376 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.555553 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.556097 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:49.556106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:49.556414 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.556625 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.556997 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":130,"threads_started":1}
I20260620 19:02:49.559147 16994 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:02:49.559427 16994 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:49.559890 16994 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
I20260620 19:02:49.560860 16669 ts_manager.cc:194] Registered new tserver with Master: 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027)
I20260620 19:02:49.561522 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:48523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:02:49.577935 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:49.683866 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:49.684118 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:49.684192 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:49.688270 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:49.688339 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:49.688423 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:02:49.693140 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:49.694036 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:49.694932 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:49.703891 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.703907 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:49.703907 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:49.703979 16999 server_base.cc:1034] running on GCE node
I20260620 19:02:49.704509 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:49.705071 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:49.706246 16999 hybrid_clock.cc:648] HybridClock initialized: now 1781982169706206 us; error 61 us; skew 500 ppm
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:49.708978 16999 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:02:49.709776 16999 webserver.cc:460] Webserver started at http://127.2.208.131:40507/ using document root <none> and password file <none>
I20260620 19:02:49.710316 16999 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:49.710402 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:49.710605 16999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:02:49.712898 16999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "1b02d027c25944eab6fac83ba4bb67a3"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "9114ed42fa1ec11d3a95aa1d780626e7"
server_key_iv: "5ea30f5bded16c74582ba4ce949305a5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.713418 16999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "1b02d027c25944eab6fac83ba4bb67a3"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "9114ed42fa1ec11d3a95aa1d780626e7"
server_key_iv: "5ea30f5bded16c74582ba4ce949305a5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.716691 16999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:02:49.718909 17014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.720067 16999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:49.720187 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "1b02d027c25944eab6fac83ba4bb67a3"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "9114ed42fa1ec11d3a95aa1d780626e7"
server_key_iv: "5ea30f5bded16c74582ba4ce949305a5"
server_key_version: "encryptionkey@0"
I20260620 19:02:49.720295 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:49.737699 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:49.740715 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:49.740898 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:49.741413 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:49.742148 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:49.742213 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.742278 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:49.742327 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:49.750782 16999 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:37105
I20260620 19:02:49.750808 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:37105 every 8 connection(s)
I20260620 19:02:49.751626 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:02:49.753464 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 16999
I20260620 19:02:49.753597 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:02:49.753883 2882 external_mini_cluster.cc:1428] Setting key bb3ec768d034eb3710bf8037522c0ccd
Jun 20 19:02:49 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:49.764595 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.753327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:55397 (local address 127.2.208.190:41125)
0620 19:02:49.753458 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.753462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.755223 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 19:02:49.756237 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.756240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.756287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.756364 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:49.757394 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.757913 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.758968 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.759180 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.759269 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.761514 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:49.761517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:49.761519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:49.761542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:49.762790 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.763238 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.763241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.763242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.763310 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.763680 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.763682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.763683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.763840 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:49.763917 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.764391 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.764484 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:02:49.765434 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.753729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:55397)
0620 19:02:49.755072 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.755123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.756038 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.756508 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.756513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.756816 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:49.757276 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.757287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.758021 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.758023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.758844 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.758852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.759305 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.759308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.759530 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.760028 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:49.760042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:49.761375 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:49.762891 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.762896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.762906 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.763154 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.763463 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:49.763465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:49.763466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:49.763599 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:49.763970 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:49.763975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:49.764183 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.764333 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.764576 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":138,"threads_started":1}
I20260620 19:02:49.766515 17128 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
I20260620 19:02:49.766721 17128 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:49.767259 17128 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
I20260620 19:02:49.768296 16669 ts_manager.cc:194] Registered new tserver with Master: 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
I20260620 19:02:49.768814 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:55397
I20260620 19:02:49.779055 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:02:49.788066 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.780900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.2.208.190:41125)
0620 19:02:49.781044 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.781049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.781200 (+ 151us) server_negotiation.cc:407] Connection header received
0620 19:02:49.781319 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.781321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.781357 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.781418 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:49.782047 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.782512 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.783285 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.783535 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.783612 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.784943 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:49.784946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:49.784948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:49.784968 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:49.786392 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.786795 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.786800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.786802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.786879 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:49.787153 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:49.787156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:49.787158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:49.787422 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:49.787564 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.787786 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.787888 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:02:49.790565 16669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:02:49.793502 16669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:02:49.807473 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.802500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.2.208.130:44027)
0620 19:02:49.803231 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.803241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.803261 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:02:49.803331 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.803336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.803684 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.803837 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:49.804005 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804738 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.806100 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.806810 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.806928 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.807072 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.807204 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.807292 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":132,"threads_started":1}
I20260620 19:02:49.807463 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.802115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.0.0.1:37504)
0620 19:02:49.802883 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.802897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.802989 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.803815 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.803818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.803835 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:49.803915 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.803921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804893 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.804899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.805977 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.805986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.806932 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.806934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.807023 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.807050 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.807167 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.807276 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":94,"thread_start_us":77,"threads_started":1}
I20260620 19:02:49.808518 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.801433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.0.0.1:51512)
0620 19:02:49.802331 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.802344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.802460 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.803129 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.803135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.803155 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:49.803300 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.803308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804826 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.804828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.806308 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.806319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.808165 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.808171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.808259 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.808279 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.808325 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.808377 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":120,"spinlock_wait_cycles":33408,"thread_start_us":119,"threads_started":1}
I20260620 19:02:49.808951 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.801670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.2.208.129:34857)
0620 19:02:49.802752 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.802761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.802779 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:02:49.802848 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.802851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.802990 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.803135 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:49.803465 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804676 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.807146 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.808027 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.808505 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.808657 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.808713 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.808812 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":122,"threads_started":1}
I20260620 19:02:49.808418 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.801399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37105 (local address 127.0.0.1:35486)
0620 19:02:49.802210 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:49.802258 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:49.802461 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:49.803216 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:49.803219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:49.803239 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:49.803353 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.803359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804825 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.804828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.806308 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:49.806319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.807771 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.807776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:49.807876 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.807899 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:02:49.808202 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 19:02:49.808271 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":162,"spinlock_wait_cycles":1792,"thread_start_us":83,"threads_started":1}
I20260620 19:02:49.809967 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:49.801514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.2.208.131:37105)
0620 19:02:49.802188 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:49.802203 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:49.802344 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:02:49.802649 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:49.802656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:49.802876 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:49.803005 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:49.803531 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.804664 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.806436 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:49.807234 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:49.807348 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:49.807508 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:49.809722 (+ 2214us) server_negotiation.cc:299] Negotiation successful
0620 19:02:49.809819 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":117,"threads_started":1}
I20260620 19:02:49.810593 16928 tablet_service.cc:1467] Processing CreateTablet for tablet c40f21a78af24d56b213eba2fee0e78f (DEFAULT_TABLE table=test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:49.811712 16928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40f21a78af24d56b213eba2fee0e78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:49.811750 16794 tablet_service.cc:1467] Processing CreateTablet for tablet c40f21a78af24d56b213eba2fee0e78f (DEFAULT_TABLE table=test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:49.812762 16794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40f21a78af24d56b213eba2fee0e78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:49.813268 17062 tablet_service.cc:1467] Processing CreateTablet for tablet c40f21a78af24d56b213eba2fee0e78f (DEFAULT_TABLE table=test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:49.814167 17062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40f21a78af24d56b213eba2fee0e78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:49.818578 17145 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Bootstrap starting.
I20260620 19:02:49.820531 17147 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Bootstrap starting.
I20260620 19:02:49.821198 17145 tablet_bootstrap.cc:654] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:49.822512 17145 log.cc:824] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:49.823109 17147 tablet_bootstrap.cc:654] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:49.823289 17146 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Bootstrap starting.
I20260620 19:02:49.823968 17147 log.cc:824] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:49.824497 17145 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: No bootstrap required, opened a new log
I20260620 19:02:49.824692 17145 ts_tablet_manager.cc:1397] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:02:49.825884 17146 tablet_bootstrap.cc:654] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:49.825944 17147 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: No bootstrap required, opened a new log
I20260620 19:02:49.826105 17147 ts_tablet_manager.cc:1397] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:02:49.826650 17146 log.cc:824] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:49.827708 17145 raft_consensus.cc:348] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.827925 17145 raft_consensus.cc:374] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:49.827998 17145 raft_consensus.cc:729] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c41b98d7c08499d9842c75a94f50129, State: Initialized, Role: FOLLOWER
I20260620 19:02:49.828426 17145 consensus_queue.cc:260] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.828502 17146 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: No bootstrap required, opened a new log
I20260620 19:02:49.828675 17146 ts_tablet_manager.cc:1397] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:02:49.829392 17145 ts_tablet_manager.cc:1428] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:02:49.829751 16860 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
I20260620 19:02:49.830767 17147 raft_consensus.cc:348] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.831022 17147 raft_consensus.cc:374] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:49.831120 17147 raft_consensus.cc:729] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b02d027c25944eab6fac83ba4bb67a3, State: Initialized, Role: FOLLOWER
I20260620 19:02:49.831557 17147 consensus_queue.cc:260] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.832159 17128 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
I20260620 19:02:49.832239 17146 raft_consensus.cc:348] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.832463 17147 ts_tablet_manager.cc:1428] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:02:49.832465 17146 raft_consensus.cc:374] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:49.832513 17146 raft_consensus.cc:729] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412b00997cfa46ec81b6d76b036da116, State: Initialized, Role: FOLLOWER
I20260620 19:02:49.832881 17146 consensus_queue.cc:260] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:49.833487 16994 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
I20260620 19:02:49.833732 17146 ts_tablet_manager.cc:1428] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 19:02:49.843806 16861 tablet.cc:2369] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:02:50.002560 17129 tablet.cc:2369] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:02:50.044265 16995 tablet.cc:2369] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:02:50.298980 17152 raft_consensus.cc:482] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:50.299204 17152 raft_consensus.cc:504] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.300498 17152 leader_election.cc:288] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857), 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027)
I20260620 19:02:50.304401 17151 raft_consensus.cc:482] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:50.304602 17151 raft_consensus.cc:504] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.305779 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.300944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:56865 (local address 127.2.208.129:34857)
0620 19:02:50.301083 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.301088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.301211 (+ 123us) server_negotiation.cc:407] Connection header received
0620 19:02:50.301384 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.301387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.301448 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.301523 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:50.302021 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.302808 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.304774 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.305199 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.305341 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.305392 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.305587 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.305658 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:02:50.305802 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.300743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.2.208.131:56865)
0620 19:02:50.301111 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.301128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.301263 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.301547 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.301550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.301590 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:50.301902 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.301908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.302920 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.302924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.304624 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.304636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.305335 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.305338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.305433 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.305484 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.305571 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.305647 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":319,"spinlock_wait_cycles":32512,"thread_start_us":80,"threads_started":1}
I20260620 19:02:50.305914 17151 leader_election.cc:288] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105), 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027)
I20260620 19:02:50.306056 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.301147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:59479 (local address 127.2.208.130:44027)
0620 19:02:50.301265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.301268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.301324 (+ 56us) server_negotiation.cc:407] Connection header received
0620 19:02:50.301382 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.301385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.301439 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.301521 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:50.302021 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.302805 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.304755 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.305327 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.305441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.305508 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.305837 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.305916 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:02:50.306418 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.300743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.2.208.131:59479)
0620 19:02:50.301111 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.301128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.301248 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.301547 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.301550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.301591 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:50.301901 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.301908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.302920 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.302924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.304624 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.304636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.305576 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.305580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.305693 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.305716 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.306183 (+ 467us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.306264 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":268,"spinlock_wait_cycles":8320,"thread_start_us":78,"threads_started":1}
I20260620 19:02:50.306373 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c41b98d7c08499d9842c75a94f50129" is_pre_election: true
I20260620 19:02:50.306612 16814 raft_consensus.cc:2454] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 0.
I20260620 19:02:50.307018 17015 leader_election.cc:302] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b02d027c25944eab6fac83ba4bb67a3, 3c41b98d7c08499d9842c75a94f50129; no voters:
I20260620 19:02:50.307287 17152 raft_consensus.cc:2781] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:02:50.307420 17152 raft_consensus.cc:482] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:02:50.307473 17152 raft_consensus.cc:3037] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:50.307806 16948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116" is_pre_election: true
I20260620 19:02:50.308087 16948 raft_consensus.cc:2454] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 0.
I20260620 19:02:50.308732 17152 raft_consensus.cc:504] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.309145 17152 leader_election.cc:288] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857), 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027)
I20260620 19:02:50.309379 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c41b98d7c08499d9842c75a94f50129"
I20260620 19:02:50.309499 16814 raft_consensus.cc:3037] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:50.309705 16948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116"
I20260620 19:02:50.309809 16948 raft_consensus.cc:3037] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:50.310771 16814 raft_consensus.cc:2454] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 1.
I20260620 19:02:50.311031 16948 raft_consensus.cc:2454] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 1.
I20260620 19:02:50.311061 17015 leader_election.cc:302] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b02d027c25944eab6fac83ba4bb67a3, 3c41b98d7c08499d9842c75a94f50129; no voters:
I20260620 19:02:50.311446 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.307224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59883 (local address 127.2.208.131:37105)
0620 19:02:50.307354 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.307380 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.307394 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:02:50.307435 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.307440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.307485 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.307555 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:50.307900 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.308580 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.310217 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.310809 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.310892 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.310941 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.311233 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.311293 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:02:50.311585 17152 raft_consensus.cc:2781] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:50.311755 17152 raft_consensus.cc:686] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b02d027c25944eab6fac83ba4bb67a3, State: Running, Role: LEADER
I20260620 19:02:50.311833 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.306245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37105 (local address 127.2.208.129:59883)
0620 19:02:50.306475 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.306490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.306569 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.307659 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.307663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.307680 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:50.307768 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.307774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.308711 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.308714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.310091 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.310102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.311038 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.311041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.311120 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.311138 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.311586 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.311661 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260620 19:02:50.312083 17152 consensus_queue.cc:237] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.312453 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "3c41b98d7c08499d9842c75a94f50129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b02d027c25944eab6fac83ba4bb67a3" is_pre_election: true
I20260620 19:02:50.312660 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.306611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.2.208.129:49311)
0620 19:02:50.307610 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.307623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.307877 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.309090 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.309094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.309113 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:50.309218 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.309223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.310191 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.310197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.311276 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.311284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.312298 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.312301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.312391 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.312411 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.312461 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.312536 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":99,"threads_started":1}
I20260620 19:02:50.313273 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.307409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:49311 (local address 127.2.208.130:44027)
0620 19:02:50.308367 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.308371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.308384 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:02:50.308418 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.308431 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.308483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.308546 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:50.309339 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.310035 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.311404 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.311942 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.313046 (+ 1104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.313087 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.313120 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.313176 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892}
I20260620 19:02:50.313657 16948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40f21a78af24d56b213eba2fee0e78f" candidate_uuid: "3c41b98d7c08499d9842c75a94f50129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116" is_pre_election: true
I20260620 19:02:50.313848 16948 raft_consensus.cc:2379] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c41b98d7c08499d9842c75a94f50129 in current term 1: Already voted for candidate 1b02d027c25944eab6fac83ba4bb67a3 in this term.
I20260620 19:02:50.314307 16749 leader_election.cc:302] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:02:50.314558 17151 raft_consensus.cc:2726] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:02:50.315805 16669 catalog_manager.cc:5495] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "1b02d027c25944eab6fac83ba4bb67a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } health_report { overall_health: UNKNOWN } } }
I20260620 19:02:50.362936 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.359783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.2.208.131:37105)
0620 19:02:50.359910 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.359914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.360094 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:02:50.360218 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.360220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.360268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.360349 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:02:50.360545 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.361002 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.361751 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.361942 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.362011 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.362246 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:02:50.362606 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:02:50.362687 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.362738 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.362775 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:02:50.368340 16948 raft_consensus.cc:1261] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Refusing update from remote peer 1b02d027c25944eab6fac83ba4bb67a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:50.368374 16814 raft_consensus.cc:1261] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Refusing update from remote peer 1b02d027c25944eab6fac83ba4bb67a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:50.369009 17152 consensus_queue.cc:1035] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:50.369195 17158 consensus_queue.cc:1035] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:50.383772 16668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:50.383909 16668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:50.386238 16668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Sending DeleteTablet for 3 replicas of tablet c40f21a78af24d56b213eba2fee0e78f
I20260620 19:02:50.386926 16794 tablet_service.cc:1514] Processing DeleteTablet for tablet c40f21a78af24d56b213eba2fee0e78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:50 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:51512
I20260620 19:02:50.386922 16928 tablet_service.cc:1514] Processing DeleteTablet for tablet c40f21a78af24d56b213eba2fee0e78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:50 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:37504
I20260620 19:02:50.386922 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet c40f21a78af24d56b213eba2fee0e78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:50 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:35486
I20260620 19:02:50.387516 17176 tablet_replica.cc:330] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: stopping tablet replica
I20260620 19:02:50.387625 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16731
I20260620 19:02:50.387825 17176 raft_consensus.cc:2229] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:02:50.388669 17177 tablet_replica.cc:330] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: stopping tablet replica
I20260620 19:02:50.388931 17177 raft_consensus.cc:2229] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:02:50.389216 17177 raft_consensus.cc:2258] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:50.389750 17176 raft_consensus.cc:2258] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:50.390774 17177 ts_tablet_manager.cc:1905] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:50.392946 17177 ts_tablet_manager.cc:1918] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:50.393061 17177 log.cc:1196] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/c40f21a78af24d56b213eba2fee0e78f
I20260620 19:02:50.393330 17177 ts_tablet_manager.cc:1939] T c40f21a78af24d56b213eba2fee0e78f P 412b00997cfa46ec81b6d76b036da116: Deleting consensus metadata
I20260620 19:02:50.394411 16655 catalog_manager.cc:4841] TS 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): tablet c40f21a78af24d56b213eba2fee0e78f (table test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]) successfully deleted
I20260620 19:02:50.395135 17176 ts_tablet_manager.cc:1905] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:02:50.396613 16653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:34857 (error 108)
I20260620 19:02:50.396817 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16865
I20260620 19:02:50.398125 17176 ts_tablet_manager.cc:1918] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:50.398236 17176 log.cc:1196] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/c40f21a78af24d56b213eba2fee0e78f
I20260620 19:02:50.398499 17176 ts_tablet_manager.cc:1939] T c40f21a78af24d56b213eba2fee0e78f P 1b02d027c25944eab6fac83ba4bb67a3: Deleting consensus metadata
I20260620 19:02:50.399492 16654 catalog_manager.cc:4841] TS 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): tablet c40f21a78af24d56b213eba2fee0e78f (table test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]) successfully deleted
I20260620 19:02:50.400198 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.399818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.0.0.1:51514)
0620 19:02:50.400058 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.400121 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
W20260620 19:02:50.400399 16653 catalog_manager.cc:4584] TS 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c40f21a78af24d56b213eba2fee0e78f: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34857: connect: Connection refused (error 111)
I20260620 19:02:50.403795 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16999
I20260620 19:02:50.409662 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 16637
I20260620 19:02:50.415920 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--webserver_interface=127.2.208.190
--webserver_port=39179
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:50.521340 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:50.521574 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:50.521652 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:50.525563 17181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:50.525607 17181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:50.525626 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:50.525689 17181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:50.525719 17181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:50.530740 17181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:50.531706 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:50.532696 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:50.545360 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:50.545401 17190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:50.545360 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:50.545754 17181 server_base.cc:1034] running on GCE node
I20260620 19:02:50.546314 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:50.547411 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:50.548611 17181 hybrid_clock.cc:648] HybridClock initialized: now 1781982170548597 us; error 31 us; skew 500 ppm
Jun 20 19:02:50 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:50.551338 17181 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:50.552222 17181 webserver.cc:460] Webserver started at http://127.2.208.190:39179/ using document root <none> and password file <none>
I20260620 19:02:50.552805 17181 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:50.552909 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:50.556658 17181 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:02:50.558672 17197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:50.559903 17181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:50.560045 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "993494df07cf98d4dd5984d016e939ce"
server_key_iv: "34bc972f13c66f94700a7f600cba63ed"
server_key_version: "encryptionkey@0"
I20260620 19:02:50.560631 17181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:50.592406 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:50.595165 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:50.595402 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:50.602058 17181 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:41125
I20260620 19:02:50.602349 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:41125 every 8 connection(s)
I20260620 19:02:50.603113 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:50.607267 17250 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:02:50.608865 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap starting.
I20260620 19:02:50.611539 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17181
I20260620 19:02:50.611982 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:34857
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=44373
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:50.626140 17250 log.cc:824] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:50.634161 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:02:50.634536 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap complete.
I20260620 19:02:50.638556 17250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:50.638907 17250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Initialized, Role: FOLLOWER
I20260620 19:02:50.639437 17250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:50.639572 17250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:50.639643 17250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:50.639760 17250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:02:50.641563 17250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:50.641888 17250 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfc7c2b39ef4ac6ba03417a01d35f9e; no voters:
I20260620 19:02:50.642210 17250 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:02:50.642282 17254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:02:50.642509 17254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 LEADER]: Becoming Leader. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Running, Role: LEADER
I20260620 19:02:50.642805 17254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:50.643446 17250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:50.643743 17254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfc7c2b39ef4ac6ba03417a01d35f9e. Latest consensus state: current_term: 2 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:50.643863 17254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:50.644217 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:50.644366 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:50.645394 17258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:50.650266 17258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:50.650851 17258 catalog_manager.cc:1259] Loaded cluster ID: 606141f66368460394a74a57ea3ab283
I20260620 19:02:50.650910 17258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:02:50.652289 17258 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:02:50.656951 17258 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:02:50.661912 17258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:02:50.662582 17258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Loaded TSK: 0
I20260620 19:02:50.663740 17258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:02:50.724541 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:50.724782 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:50.724845 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:50.728842 17252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:50.728899 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:50.728982 17252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:02:50.734069 17252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:34857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=44373
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:50.735041 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:50.736052 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:50.746448 17277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:50.746448 17278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:50.746744 17252 server_base.cc:1034] running on GCE node
W20260620 19:02:50.746883 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:50.747239 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:50.747897 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:50.749073 17252 hybrid_clock.cc:648] HybridClock initialized: now 1781982170749062 us; error 27 us; skew 500 ppm
Jun 20 19:02:50 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:50.751868 17252 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:02:50.752744 17252 webserver.cc:460] Webserver started at http://127.2.208.129:44373/ using document root <none> and password file <none>
I20260620 19:02:50.753290 17252 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:50.753393 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:50.757346 17252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:02:50.759701 17287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:50.760969 17252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:02:50.761099 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "3c41b98d7c08499d9842c75a94f50129"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "05650ab1833b4c7f1b6bd19487de83c0"
server_key_iv: "c655451df76e322671d1e2da6e52907f"
server_key_version: "encryptionkey@0"
I20260620 19:02:50.761649 17252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:50.802176 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:50.805099 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:50.805298 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:50.805883 17252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:50.806962 17294 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:02:50.811226 17252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:02:50.811386 17252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 19:02:50.811483 17252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:02:50.813158 17252 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:02:50.813234 17252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:02:50.813480 17294 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Bootstrap starting.
I20260620 19:02:50.823217 17252 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:34857
I20260620 19:02:50.823297 17401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:34857 every 8 connection(s)
I20260620 19:02:50.824196 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:02:50.828778 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17252
I20260620 19:02:50.829267 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:44027
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=45633
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:50.833173 17294 log.cc:824] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:02:50 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:50.841538 17294 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:02:50.841979 17294 tablet_bootstrap.cc:492] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Bootstrap complete.
I20260620 19:02:50.843207 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.826490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:46265 (local address 127.2.208.190:41125)
0620 19:02:50.827067 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.827086 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.827134 (+ 48us) server_negotiation.cc:407] Connection header received
0620 19:02:50.828355 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.828375 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.828814 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.829191 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:50.830934 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.831994 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.833135 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.833878 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.834068 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.837216 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:50.837222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:50.837238 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:50.837271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:50.839442 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:50.840146 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:50.840155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:50.840160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:50.840325 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:50.840724 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:50.840731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:50.840736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:50.841256 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:50.841409 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.842110 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.842406 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":140,"threads_started":1}
I20260620 19:02:50.843233 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.826454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:46265)
0620 19:02:50.826928 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.826981 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.827866 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.829267 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.829277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.830041 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:50.830712 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.830735 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.832153 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.832156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.833009 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.833018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.834036 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.834041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.834204 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.834898 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:50.834938 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:50.836991 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:50.839570 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:50.839579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:50.839597 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:50.839992 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:50.840442 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:50.840445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:50.840447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:50.840612 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:50.841391 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:50.841397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:50.841673 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.841896 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.842228 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260620 19:02:50.843317 17294 ts_tablet_manager.cc:1397] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.000s
I20260620 19:02:50.846063 17402 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
I20260620 19:02:50.846333 17402 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:50.846086 17294 raft_consensus.cc:348] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.846639 17294 raft_consensus.cc:729] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c41b98d7c08499d9842c75a94f50129, State: Initialized, Role: FOLLOWER
I20260620 19:02:50.846871 17402 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
I20260620 19:02:50.847100 17294 consensus_queue.cc:260] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } } peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } }
I20260620 19:02:50.848151 17294 ts_tablet_manager.cc:1428] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:02:50.849730 17213 ts_manager.cc:194] Registered new tserver with Master: 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857)
I20260620 19:02:50.854488 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:46265
I20260620 19:02:50.856791 17402 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
Jun 20 19:02:50 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.129@KRBTEST.COM
I20260620 19:02:50.863986 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.853174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.0.0.1:51518)
0620 19:02:50.853503 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:50.853536 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:50.853688 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:50.854131 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:50.854136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:50.854478 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:50.854605 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.854614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.855854 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.855857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.856785 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:50.856796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.857769 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.857772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:50.857882 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.858534 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:50.858549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:50.860102 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:50.862205 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:50.862212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:50.862216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:50.862474 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:50.862795 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:50.862797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:50.862798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:50.862874 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:50.863463 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:50.863468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:50.863587 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 19:02:50.863704 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:02:50.863850 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260620 19:02:50.864457 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:50.852916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.2.208.129:34857)
0620 19:02:50.853189 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:50.853197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:50.853622 (+ 425us) server_negotiation.cc:407] Connection header received
0620 19:02:50.853827 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:50.853833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:50.853992 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:50.854130 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:50.854714 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.855723 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.856926 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:50.857516 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:50.857607 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:50.860241 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:50.860251 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:50.860262 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:50.860287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:50.862092 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:50.862600 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:50.862606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:50.862608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:50.862702 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:50.862998 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:50.863003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:50.863007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:50.863285 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:50.863445 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:50.863847 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 19:02:50.864019 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260620 19:02:50.865450 17336 tablet_service.cc:1514] Processing DeleteTablet for tablet c40f21a78af24d56b213eba2fee0e78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:50 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:51518
I20260620 19:02:50.866372 17411 tablet_replica.cc:330] stopping tablet replica
I20260620 19:02:50.866621 17411 raft_consensus.cc:2229] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:02:50.866904 17411 raft_consensus.cc:2258] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:50.867863 17411 ts_tablet_manager.cc:1905] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:50.871150 17411 ts_tablet_manager.cc:1918] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:50.871241 17411 log.cc:1196] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/c40f21a78af24d56b213eba2fee0e78f
I20260620 19:02:50.871541 17411 ts_tablet_manager.cc:1939] T c40f21a78af24d56b213eba2fee0e78f P 3c41b98d7c08499d9842c75a94f50129: Deleting consensus metadata
I20260620 19:02:50.873121 17198 catalog_manager.cc:4841] TS 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857): tablet c40f21a78af24d56b213eba2fee0e78f (table test-table [id=c33a16e2c6844c59bd2ba65f15cbfd48]) successfully deleted
W20260620 19:02:50.950573 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:50.950795 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:50.950835 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:50.954738 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:50.954787 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:50.954840 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:02:50.959959 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:44027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=45633
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:50.960810 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:50.961687 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:50.971120 17416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:50.971156 17417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:50.971120 17419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:50.971264 17406 server_base.cc:1034] running on GCE node
I20260620 19:02:50.971921 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:50.972522 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:50.973676 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781982170973662 us; error 27 us; skew 500 ppm
Jun 20 19:02:50 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:50.976262 17406 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:02:50.977167 17406 webserver.cc:460] Webserver started at http://127.2.208.130:45633/ using document root <none> and password file <none>
I20260620 19:02:50.977705 17406 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:50.977797 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:50.981639 17406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:02:50.983603 17426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:50.984757 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:50.984870 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "412b00997cfa46ec81b6d76b036da116"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "af0323662b96cb61de01e1e5e6f5c240"
server_key_iv: "4ad1c17034a0b8266d384899294fd1c5"
server_key_version: "encryptionkey@0"
I20260620 19:02:50.985426 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:50.996073 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:50.998660 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:50.998859 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:50.999471 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:51.000227 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:51.000272 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.000341 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:51.000382 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.009287 17406 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44027
I20260620 19:02:51.009313 17539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44027 every 8 connection(s)
I20260620 19:02:51.010206 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:02:51.017722 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17406
I20260620 19:02:51.018146 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:37105
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=40507
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:51.023707 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.011945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:58335 (local address 127.2.208.190:41125)
0620 19:02:51.012080 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.012084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.013019 (+ 935us) server_negotiation.cc:407] Connection header received
0620 19:02:51.013850 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.013852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.013897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.013973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:51.015284 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.015864 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.016911 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.017156 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.017231 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.019888 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:51.019894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:51.019899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:51.019931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:51.021505 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.022100 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.022103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.022104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.022178 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.022582 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.022584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.022585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.022742 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:51.022815 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.023492 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.023591 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:02:51.024906 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.012312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:58335)
0620 19:02:51.012886 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.012922 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.013681 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.014219 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.014225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.014691 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:51.015153 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.015166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.015970 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.015972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.016812 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.016821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.017276 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.017278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.017427 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.018001 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:51.018016 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:51.019705 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:51.021620 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:51.021629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:51.021645 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:51.022001 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:51.022272 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:51.022279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:51.022284 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:51.022486 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:51.022850 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:51.022858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:51.023178 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.023449 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.023793 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":95,"threads_started":1}
I20260620 19:02:51.026125 17540 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
I20260620 19:02:51.026343 17540 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:51.026829 17540 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
I20260620 19:02:51.027905 17213 ts_manager.cc:194] Registered new tserver with Master: 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027)
I20260620 19:02:51.028455 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:58335
W20260620 19:02:51.141469 17543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:51.141738 17543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:51.141805 17543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:51.146462 17543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:51.146539 17543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:51.146658 17543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:02:51.151625 17543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:37105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=40507
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:51.152515 17543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:51.153473 17543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:51.162601 17549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.162675 17548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.162801 17551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:51.162838 17543 server_base.cc:1034] running on GCE node
I20260620 19:02:51.163403 17543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:51.163980 17543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:51.165158 17543 hybrid_clock.cc:648] HybridClock initialized: now 1781982171165133 us; error 44 us; skew 500 ppm
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:51.167923 17543 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:02:51.168731 17543 webserver.cc:460] Webserver started at http://127.2.208.131:40507/ using document root <none> and password file <none>
I20260620 19:02:51.169261 17543 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:51.169330 17543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:51.172843 17543 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:51.174808 17558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.175977 17543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:51.176102 17543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "1b02d027c25944eab6fac83ba4bb67a3"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "9114ed42fa1ec11d3a95aa1d780626e7"
server_key_iv: "5ea30f5bded16c74582ba4ce949305a5"
server_key_version: "encryptionkey@0"
I20260620 19:02:51.176604 17543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:51.186587 17543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:51.189793 17543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:51.190002 17543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:51.190550 17543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:51.191349 17543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:51.191444 17543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.191505 17543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:51.191545 17543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.199523 17543 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:37105
I20260620 19:02:51.199574 17671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:37105 every 8 connection(s)
I20260620 19:02:51.200469 17543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:02:51.205152 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17543
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:02:51.213972 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.205786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.2.208.190:41125)
0620 19:02:51.206067 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.206071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.206181 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:02:51.206298 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.206301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.206344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.206436 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:51.207115 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.207821 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.208636 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.208991 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.209098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.210456 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:51.210460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:51.210463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:51.210485 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:51.212287 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.212702 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.212706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.212707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.212791 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.213093 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.213096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.213098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.213314 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:51.213502 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.213692 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.213803 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260620 19:02:51.215401 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.202509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:36981)
0620 19:02:51.203084 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.203121 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.203943 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.204466 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.204473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.204783 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:51.205235 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.205255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.206179 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.206183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.207309 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.207320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.207817 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.207821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.208017 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.208627 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:51.208647 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:51.210460 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:51.212499 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:51.212503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:51.212514 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:51.212821 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:51.213093 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:51.213096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:51.213098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:51.213270 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:51.213663 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:51.213669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:51.213951 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.214146 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.214457 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":104,"threads_started":1}
I20260620 19:02:51.215793 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.202165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:36981 (local address 127.2.208.190:41125)
0620 19:02:51.202291 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.202295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.203216 (+ 921us) server_negotiation.cc:407] Connection header received
0620 19:02:51.204134 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.204137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.204194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.204293 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:51.205382 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.206018 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.207464 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.207674 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.207751 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.210617 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:51.210620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:51.210621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:51.210653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:51.212391 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.212910 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.212913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.212916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.212997 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:51.213379 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:51.213382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:51.213385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:51.213562 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:51.213648 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.215515 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.215626 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:02:51.216506 17672 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41125
I20260620 19:02:51.216707 17672 heartbeater.cc:464] Registering TS with master...
I20260620 19:02:51.217118 17672 heartbeater.cc:511] Master 127.2.208.190:41125 requested a full tablet report, sending...
I20260620 19:02:51.218003 17213 ts_manager.cc:194] Registered new tserver with Master: 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
I20260620 19:02:51.218479 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:36981
I20260620 19:02:51.227252 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:02:51.231559 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.227928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.2.208.190:41125)
0620 19:02:51.228049 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.228055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.228243 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:02:51.228419 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.228421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.228507 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.228614 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:02:51.228846 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.229326 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.230204 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.230420 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.230527 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.230827 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:02:51.231167 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:02:51.231292 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.231353 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.231433 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:02:51.232188 17213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:02:51.235229 17213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:02:51.242651 17336 tablet_service.cc:1467] Processing CreateTablet for tablet eb2e18540708432f8ff32926bbe9624a (DEFAULT_TABLE table=test-table [id=ebdf482987474dd9a0d5405f0574f2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:51.243292 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2e18540708432f8ff32926bbe9624a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:51.246325 17294 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Bootstrap starting.
I20260620 19:02:51.246744 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.241502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.0.0.1:37518)
0620 19:02:51.241631 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.241646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.241727 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.242507 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.242510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.242523 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.242620 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.242625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.244141 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.244144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.245397 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.245408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.246305 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.246308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.246428 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.246474 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.246528 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.246598 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":25}
I20260620 19:02:51.247057 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.242105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.2.208.131:37105)
0620 19:02:51.242539 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.242550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.242572 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:02:51.242643 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.242648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.242841 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.243000 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.243354 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.244515 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.245706 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.246485 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.246613 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.246747 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.246801 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.246906 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":150,"threads_started":1}
I20260620 19:02:51.247262 17294 tablet_bootstrap.cc:654] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:51.247411 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.241776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37105 (local address 127.0.0.1:35492)
0620 19:02:51.242292 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.242304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.242383 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.242981 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.242984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.243003 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.243217 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.243225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.244668 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.244673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.245603 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.245613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.246612 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.246615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.246693 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.246713 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.246774 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.247031 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":23,"thread_start_us":71,"threads_started":1}
I20260620 19:02:51.248054 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.241672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.2.208.130:44027)
0620 19:02:51.242018 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.242034 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.242059 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:02:51.242132 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.242137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.242377 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.242531 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.242732 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.243974 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.245524 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.246146 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.247403 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.247705 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.247770 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.247876 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":126,"threads_started":1}
I20260620 19:02:51.250443 17606 tablet_service.cc:1467] Processing CreateTablet for tablet eb2e18540708432f8ff32926bbe9624a (DEFAULT_TABLE table=test-table [id=ebdf482987474dd9a0d5405f0574f2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:51.250443 17474 tablet_service.cc:1467] Processing CreateTablet for tablet eb2e18540708432f8ff32926bbe9624a (DEFAULT_TABLE table=test-table [id=ebdf482987474dd9a0d5405f0574f2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:02:51.251552 17606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2e18540708432f8ff32926bbe9624a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:51.251549 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2e18540708432f8ff32926bbe9624a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:02:51.251806 17294 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: No bootstrap required, opened a new log
I20260620 19:02:51.251909 17294 ts_tablet_manager.cc:1397] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:02:51.252293 17294 raft_consensus.cc:348] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.252419 17294 raft_consensus.cc:374] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:51.252465 17294 raft_consensus.cc:729] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c41b98d7c08499d9842c75a94f50129, State: Initialized, Role: FOLLOWER
I20260620 19:02:51.252586 17294 consensus_queue.cc:260] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.252876 17294 ts_tablet_manager.cc:1428] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:02:51.259774 17686 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Bootstrap starting.
I20260620 19:02:51.261994 17686 tablet_bootstrap.cc:654] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:51.262378 17685 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Bootstrap starting.
I20260620 19:02:51.262728 17686 log.cc:824] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:51.264441 17686 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: No bootstrap required, opened a new log
I20260620 19:02:51.264602 17685 tablet_bootstrap.cc:654] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Neither blocks nor log segments found. Creating new log.
I20260620 19:02:51.264678 17686 ts_tablet_manager.cc:1397] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:02:51.265430 17685 log.cc:824] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:51.267088 17685 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: No bootstrap required, opened a new log
I20260620 19:02:51.267270 17685 ts_tablet_manager.cc:1397] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:02:51.268018 17686 raft_consensus.cc:348] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.268218 17686 raft_consensus.cc:374] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:51.268252 17686 raft_consensus.cc:729] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b02d027c25944eab6fac83ba4bb67a3, State: Initialized, Role: FOLLOWER
I20260620 19:02:51.268666 17686 consensus_queue.cc:260] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.269246 17672 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
I20260620 19:02:51.269456 17686 ts_tablet_manager.cc:1428] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 19:02:51.271677 17685 raft_consensus.cc:348] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.271935 17685 raft_consensus.cc:374] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:02:51.271998 17685 raft_consensus.cc:729] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412b00997cfa46ec81b6d76b036da116, State: Initialized, Role: FOLLOWER
I20260620 19:02:51.272410 17685 consensus_queue.cc:260] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.272985 17540 heartbeater.cc:503] Master 127.2.208.190:41125 was elected leader, sending a full tablet report...
I20260620 19:02:51.273221 17685 ts_tablet_manager.cc:1428] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:02:51.300280 17689 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:51.300483 17689 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.301614 17689 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857)
I20260620 19:02:51.303161 17408 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:51.303313 17408 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.304478 17408 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
I20260620 19:02:51.306053 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.301773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.2.208.131:33871)
0620 19:02:51.301902 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.301918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.302014 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.302407 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.302409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.302423 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.302546 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.302552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.303556 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.303560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.304561 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.304576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.305641 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.305644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.305742 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.305764 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.305828 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.305899 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":43}
I20260620 19:02:51.306437 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.301969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:36453 (local address 127.2.208.129:34857)
0620 19:02:51.302096 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.302102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.302284 (+ 182us) server_negotiation.cc:407] Connection header received
0620 19:02:51.302370 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.302373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.302440 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.302536 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.302831 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.303601 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.305058 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.305702 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.305779 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.305908 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.306255 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.306329 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:02:51.306648 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.301998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:33871 (local address 127.2.208.130:44027)
0620 19:02:51.302160 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.302165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.302179 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:02:51.302235 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.302237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.302312 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.302383 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.302664 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.303411 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.304747 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.305418 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.305535 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.305591 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.306441 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.306511 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 19:02:51.306809 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.301859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.2.208.131:36453)
0620 19:02:51.302177 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.302191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.302286 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.302566 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.302568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.302584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.302703 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.302709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.303717 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.303721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.304919 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.304930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.306014 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.306020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.306122 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.306143 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.306600 (+ 457us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.306664 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":21,"thread_start_us":98,"threads_started":1}
I20260620 19:02:51.307178 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116" is_pre_election: true
I20260620 19:02:51.307262 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c41b98d7c08499d9842c75a94f50129" is_pre_election: true
I20260620 19:02:51.307487 17494 raft_consensus.cc:2454] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 0.
I20260620 19:02:51.307508 17356 raft_consensus.cc:2454] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 0.
I20260620 19:02:51.307910 17559 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b02d027c25944eab6fac83ba4bb67a3, 3c41b98d7c08499d9842c75a94f50129; no voters:
I20260620 19:02:51.308189 17689 raft_consensus.cc:2781] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:02:51.308275 17689 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:02:51.308329 17689 raft_consensus.cc:3037] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:51.309348 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.304654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.2.208.129:46301)
0620 19:02:51.304874 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.304887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.304968 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.305746 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.305749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.305765 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.305834 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.305838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.306894 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.306897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.308063 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.308074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.308924 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.308927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.309020 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.309045 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.309120 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.309202 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 19:02:51.309376 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.305107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:46301 (local address 127.2.208.130:44027)
0620 19:02:51.305532 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.305540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.305551 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:02:51.305588 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.305591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.305628 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.305924 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.306045 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.306763 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.308199 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.308793 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.308899 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.308936 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.309159 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.309243 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":109,"threads_started":1}
I20260620 19:02:51.309648 17689 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.309808 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "3c41b98d7c08499d9842c75a94f50129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116" is_pre_election: true
I20260620 19:02:51.309921 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.305052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59801 (local address 127.2.208.131:37105)
0620 19:02:51.305352 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.305356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.305370 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:02:51.305419 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.305422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.305469 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.305533 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:02:51.305861 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.306532 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.308603 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.309116 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.309238 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.309276 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.309749 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.309821 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260620 19:02:51.309929 17494 raft_consensus.cc:2454] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c41b98d7c08499d9842c75a94f50129 in term 0.
I20260620 19:02:51.309998 17689 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 election: Requested vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857)
I20260620 19:02:51.310267 17290 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129, 412b00997cfa46ec81b6d76b036da116; no voters:
I20260620 19:02:51.310408 17356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c41b98d7c08499d9842c75a94f50129"
I20260620 19:02:51.310527 17356 raft_consensus.cc:3037] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:51.310575 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.304745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37105 (local address 127.2.208.129:59801)
0620 19:02:51.305223 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.305236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.305325 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.305632 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:51.305635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:51.305652 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:02:51.305745 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.305751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.306651 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.306656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.308464 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:51.308474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.309345 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.309349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:51.309607 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.309630 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:02:51.309733 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:02:51.309854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":455,"thread_start_us":88,"threads_started":1}
I20260620 19:02:51.311044 17626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "3c41b98d7c08499d9842c75a94f50129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b02d027c25944eab6fac83ba4bb67a3" is_pre_election: true
I20260620 19:02:51.311264 17626 raft_consensus.cc:2379] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c41b98d7c08499d9842c75a94f50129 in current term 1: Already voted for candidate 1b02d027c25944eab6fac83ba4bb67a3 in this term.
I20260620 19:02:51.311740 17356 raft_consensus.cc:2454] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 1.
I20260620 19:02:51.311879 17408 raft_consensus.cc:2741] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 19:02:51.312005 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2e18540708432f8ff32926bbe9624a" candidate_uuid: "1b02d027c25944eab6fac83ba4bb67a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412b00997cfa46ec81b6d76b036da116"
I20260620 19:02:51.312071 17559 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b02d027c25944eab6fac83ba4bb67a3, 3c41b98d7c08499d9842c75a94f50129; no voters:
I20260620 19:02:51.312134 17494 raft_consensus.cc:3037] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:02:51.312242 17689 raft_consensus.cc:2781] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:02:51.312450 17689 raft_consensus.cc:686] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b02d027c25944eab6fac83ba4bb67a3, State: Running, Role: LEADER
I20260620 19:02:51.312753 17689 consensus_queue.cc:237] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.313298 17494 raft_consensus.cc:2454] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b02d027c25944eab6fac83ba4bb67a3 in term 1.
I20260620 19:02:51.315955 17214 catalog_manager.cc:5495] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "1b02d027c25944eab6fac83ba4bb67a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } health_report { overall_health: UNKNOWN } } }
W20260620 19:02:51.325344 17403 tablet.cc:2369] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:02:51.345602 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.341915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.2.208.131:37105)
0620 19:02:51.342053 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:51.342057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:51.342069 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:02:51.342217 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:51.342219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:51.342274 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:51.342389 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:02:51.342658 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.343139 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.344161 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:51.344373 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:51.344480 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:51.344785 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:02:51.345219 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:02:51.345345 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:51.345391 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:02:51.345439 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:02:51.351125 17356 raft_consensus.cc:1261] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Refusing update from remote peer 1b02d027c25944eab6fac83ba4bb67a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:51.351140 17494 raft_consensus.cc:1261] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Refusing update from remote peer 1b02d027c25944eab6fac83ba4bb67a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:02:51.351781 17694 consensus_queue.cc:1035] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:51.351960 17689 consensus_queue.cc:1035] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:02:51.358060 17697 mvcc.cc:205] Tried to move back new op lower bound from 7298998973847752704 to 7298998973699739648. Current Snapshot: MvccSnapshot[applied={T|T < 7298998973847752704}]
I20260620 19:02:51.366041 17214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:51.366225 17214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:02:51.368530 17214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Sending DeleteTablet for 3 replicas of tablet eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:51.369141 17474 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2e18540708432f8ff32926bbe9624a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:51 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:37518
I20260620 19:02:51.369270 17336 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2e18540708432f8ff32926bbe9624a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:51 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:51518
I20260620 19:02:51.369350 17606 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2e18540708432f8ff32926bbe9624a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:02:51 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:35492
I20260620 19:02:51.369604 17706 tablet_replica.cc:330] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: stopping tablet replica
I20260620 19:02:51.369716 17411 tablet_replica.cc:330] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: stopping tablet replica
I20260620 19:02:51.369814 17411 raft_consensus.cc:2229] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:02:51.369885 17706 raft_consensus.cc:2229] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:02:51.369936 17411 raft_consensus.cc:2258] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:51.369936 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17252
I20260620 19:02:51.370151 17706 raft_consensus.cc:2258] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:51.371002 17707 tablet_replica.cc:330] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: stopping tablet replica
I20260620 19:02:51.371294 17707 raft_consensus.cc:2229] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:02:51.371671 17707 raft_consensus.cc:2258] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:02:51.373143 17707 ts_tablet_manager.cc:1905] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:51.375921 17707 ts_tablet_manager.cc:1918] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:51.376016 17707 log.cc:1196] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:51.376358 17707 ts_tablet_manager.cc:1939] T eb2e18540708432f8ff32926bbe9624a P 1b02d027c25944eab6fac83ba4bb67a3: Deleting consensus metadata
I20260620 19:02:51.376861 17706 ts_tablet_manager.cc:1905] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:02:51.377063 17198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:34857 (error 108)
I20260620 19:02:51.377626 17199 catalog_manager.cc:4841] TS 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): tablet eb2e18540708432f8ff32926bbe9624a (table test-table [id=ebdf482987474dd9a0d5405f0574f2dd]) successfully deleted
I20260620 19:02:51.378669 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17406
I20260620 19:02:51.378775 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.378453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34857 (local address 127.0.0.1:51528)
0620 19:02:51.378620 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.378689 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:02:51.383775 17198 catalog_manager.cc:4584] TS 3c41b98d7c08499d9842c75a94f50129 (127.2.208.129:34857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb2e18540708432f8ff32926bbe9624a: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34857: connect: Connection refused (error 111)
I20260620 19:02:51.384672 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17543
I20260620 19:02:51.384970 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.384704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.0.0.1:37532)
0620 19:02:51.384831 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.384886 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.130:44027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 19:02:51.390452 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17181
I20260620 19:02:51.396667 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--webserver_interface=127.2.208.190
--webserver_port=39179
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:51.504325 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:51.504551 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:51.504586 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:51.508433 17709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:02:51.508490 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:51.508508 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:51.508574 17709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:02:51.508603 17709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:02:51.513423 17709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:51.514345 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:51.515244 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:02:51.523255 17709 server_base.cc:1034] running on GCE node
W20260620 19:02:51.523233 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.523309 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.523250 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:51.524192 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:51.525187 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:51.526362 17709 hybrid_clock.cc:648] HybridClock initialized: now 1781982171526335 us; error 41 us; skew 500 ppm
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:51.528946 17709 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:02:51.529675 17709 webserver.cc:460] Webserver started at http://127.2.208.190:39179/ using document root <none> and password file <none>
I20260620 19:02:51.530146 17709 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:51.530200 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:51.534201 17709 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:02:51.536126 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.537250 17709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 19:02:51.537375 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "993494df07cf98d4dd5984d016e939ce"
server_key_iv: "34bc972f13c66f94700a7f600cba63ed"
server_key_version: "encryptionkey@0"
I20260620 19:02:51.537866 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:51.565692 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:51.568779 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:51.568984 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:51.575816 17709 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:41125
I20260620 19:02:51.576069 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:41125 every 8 connection(s)
I20260620 19:02:51.576861 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:02:51.580996 17777 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:02:51.582383 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17709
I20260620 19:02:51.582707 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap starting.
I20260620 19:02:51.582841 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:34857
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=44373
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:51.600003 17777 log.cc:824] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:51.610639 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:02:51.610987 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e: Bootstrap complete.
I20260620 19:02:51.614878 17777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:51.615262 17777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Initialized, Role: FOLLOWER
I20260620 19:02:51.615841 17777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:51.616003 17777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:02:51.616081 17777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:02:51.616191 17777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:02:51.618037 17777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:51.618347 17777 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfc7c2b39ef4ac6ba03417a01d35f9e; no voters:
I20260620 19:02:51.618670 17777 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:02:51.618747 17781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:02:51.618938 17781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [term 3 LEADER]: Becoming Leader. State: Replica: 9dfc7c2b39ef4ac6ba03417a01d35f9e, State: Running, Role: LEADER
I20260620 19:02:51.619210 17781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } }
I20260620 19:02:51.619853 17777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:02:51.620128 17781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfc7c2b39ef4ac6ba03417a01d35f9e. Latest consensus state: current_term: 3 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:51.620249 17781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:51.620591 17785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:02:51.620589 17782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfc7c2b39ef4ac6ba03417a01d35f9e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41125 } } }
I20260620 19:02:51.620829 17782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfc7c2b39ef4ac6ba03417a01d35f9e [sys.catalog]: This master's current role is: LEADER
I20260620 19:02:51.625134 17785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:02:51.625742 17785 catalog_manager.cc:1259] Loaded cluster ID: 606141f66368460394a74a57ea3ab283
I20260620 19:02:51.625802 17785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:02:51.626835 17785 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5204869ad google::LogMessage::Fail() at ??:0
@ 0x7fc52048988a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5204864de google::LogMessage::Flush() at ??:0
@ 0x7fc520487a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc52adc5fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc52adc6d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc52adc3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc52ae07640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5226d83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5226d8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc5226da3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5226c9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc523d0d6db start_thread at ??:0
@ 0x7fc51ef8461f clone at ??:0
W20260620 19:02:51.717059 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:51.717284 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:51.717329 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:51.721320 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:51.721369 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:51.721450 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:02:51.726509 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:34857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=44373
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:51.727559 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:51.728535 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:51.737705 17819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.737705 17818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.737715 17821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:51.737928 17779 server_base.cc:1034] running on GCE node
I20260620 19:02:51.738325 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:51.739007 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:51.740187 17779 hybrid_clock.cc:648] HybridClock initialized: now 1781982171740164 us; error 35 us; skew 500 ppm
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:51.742722 17779 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:02:51.743613 17779 webserver.cc:460] Webserver started at http://127.2.208.129:44373/ using document root <none> and password file <none>
I20260620 19:02:51.744124 17779 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:51.744180 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:51.747788 17779 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 19:02:51.749979 17828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.751255 17779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:51.751402 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "3c41b98d7c08499d9842c75a94f50129"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "05650ab1833b4c7f1b6bd19487de83c0"
server_key_iv: "c655451df76e322671d1e2da6e52907f"
server_key_version: "encryptionkey@0"
I20260620 19:02:51.751895 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:51.787572 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:51.790827 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:51.791036 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:51.791651 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:51.792766 17835 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:02:51.798692 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:02:51.798784 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 19:02:51.798867 17779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:02:51.800567 17779 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:02:51.800631 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 19:02:51.800968 17835 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Bootstrap starting.
I20260620 19:02:51.810985 17779 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:34857
I20260620 19:02:51.811048 17942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:34857 every 8 connection(s)
I20260620 19:02:51.812018 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:02:51.819763 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17779
I20260620 19:02:51.820165 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:44027
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=45633
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:02:51.825327 17835 log.cc:824] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Log is configured to *not* fsync() on all Append() calls
I20260620 19:02:51.832064 17835 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:02:51.832413 17835 tablet_bootstrap.cc:492] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Bootstrap complete.
I20260620 19:02:51.833518 17835 ts_tablet_manager.cc:1397] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20260620 19:02:51.835887 17835 raft_consensus.cc:348] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.836287 17835 raft_consensus.cc:729] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c41b98d7c08499d9842c75a94f50129, State: Initialized, Role: FOLLOWER
I20260620 19:02:51.836755 17835 consensus_queue.cc:260] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:51.837661 17835 ts_tablet_manager.cc:1428] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260620 19:02:51.885896 17945 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:02:51.813908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:46941)
0620 19:02:51.814397 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.814479 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:51.815283 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:51.885150 (+ 69867us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
W20260620 19:02:51.886678 17943 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:41125 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:41125: Client connection negotiation failed: client connection to 127.2.208.190:41125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:02:51.887452 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.887141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:59653)
0620 19:02:51.887265 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.887342 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:02:51.888177 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:51.887944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:56343)
0620 19:02:51.888064 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:51.888121 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 19:02:51.888397 17943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:02:51.951807 17946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:51.952034 17946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:51.952092 17946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:51.956148 17946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:51.956197 17946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:51.956286 17946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:02:51.961342 17946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:44027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=45633
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:51.962318 17946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:51.963296 17946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:51.972535 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:51.972551 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:51.972644 17946 server_base.cc:1034] running on GCE node
W20260620 19:02:51.972551 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:51.973094 17946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:51.973698 17946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:51.974857 17946 hybrid_clock.cc:648] HybridClock initialized: now 1781982171974841 us; error 26 us; skew 500 ppm
Jun 20 19:02:51 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:51.977651 17946 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:02:51.978505 17946 webserver.cc:460] Webserver started at http://127.2.208.130:45633/ using document root <none> and password file <none>
I20260620 19:02:51.979022 17946 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:51.979107 17946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:51.982817 17946 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:02:51.984838 17963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:51.986016 17946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:02:51.986160 17946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "412b00997cfa46ec81b6d76b036da116"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "af0323662b96cb61de01e1e5e6f5c240"
server_key_iv: "4ad1c17034a0b8266d384899294fd1c5"
server_key_version: "encryptionkey@0"
I20260620 19:02:51.986678 17946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:51.998701 17946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:52.001777 17946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:52.001962 17946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:52.002509 17946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:52.003523 17970 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 19:02:52.007014 17970 ts_tablet_manager.cc:1852] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 19:02:52.007459 17970 ts_tablet_manager.cc:1905] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:02:52.011097 17970 ts_tablet_manager.cc:1918] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:02:52.011224 17970 log.cc:1196] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:52.011611 17970 ts_tablet_manager.cc:1939] T eb2e18540708432f8ff32926bbe9624a P 412b00997cfa46ec81b6d76b036da116: Deleting consensus metadata
I20260620 19:02:52.011910 17946 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 19:02:52.011972 17946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260620 19:02:52.012045 17946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:52.012081 17946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:52.020888 17946 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44027
I20260620 19:02:52.020941 18077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44027 every 8 connection(s)
I20260620 19:02:52.021711 17946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:02:52.024909 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.023751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:49157)
0620 19:02:52.024180 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.024280 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
W20260620 19:02:52.025373 18078 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:41125 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:41125: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
I20260620 19:02:52.025897 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.025709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:42983)
0620 19:02:52.025808 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.025850 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:02:52.026472 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.026304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:58195)
0620 19:02:52.026392 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.026430 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:02:52.026636 18078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:02:52.028254 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 17946
I20260620 19:02:52.028626 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:37105
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:41125
--webserver_port=40507
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:02:52.136595 18081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:02:52.136811 18081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:02:52.136852 18081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:02:52.140609 18081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:02:52.140648 18081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:02:52.140700 18081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:02:52.145618 18081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:37105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=40507
--tserver_master_addrs=127.2.208.190:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:02:52.146481 18081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:02:52.147401 18081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:02:52.156628 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:02:52.156690 18089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:52.156772 18081 server_base.cc:1034] running on GCE node
W20260620 19:02:52.156661 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:02:52.157177 18081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:02:52.157763 18081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:02:52.158954 18081 hybrid_clock.cc:648] HybridClock initialized: now 1781982172158924 us; error 43 us; skew 500 ppm
Jun 20 19:02:52 dist-test-slave-tsm4 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982172, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:02:52.161613 18081 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:02:52.162355 18081 webserver.cc:460] Webserver started at http://127.2.208.131:40507/ using document root <none> and password file <none>
I20260620 19:02:52.162894 18081 fs_manager.cc:359] Metadata directory not provided
I20260620 19:02:52.162958 18081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:02:52.166764 18081 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:02:52.168783 18096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:52.170020 18081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:02:52.170147 18081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "1b02d027c25944eab6fac83ba4bb67a3"
format_stamp: "Formatted at 2026-06-20 19:02:49 on dist-test-slave-tsm4"
server_key: "9114ed42fa1ec11d3a95aa1d780626e7"
server_key_iv: "5ea30f5bded16c74582ba4ce949305a5"
server_key_version: "encryptionkey@0"
I20260620 19:02:52.170653 18081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:02:52.189877 18081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:02:52.192747 18081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:02:52.192952 18081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:02:52.193547 18081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:02:52.194352 18081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:02:52.194396 18081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:52.194463 18081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:02:52.194496 18081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:02:52.203287 18081 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:37105
I20260620 19:02:52.203308 18209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:37105 every 8 connection(s)
I20260620 19:02:52.204262 18081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:02:52.204589 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18081
I20260620 19:02:52.208415 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.206947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:54245)
0620 19:02:52.207643 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.207742 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":120,"threads_started":1}
W20260620 19:02:52.208814 18210 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:41125 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:41125: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
I20260620 19:02:52.209393 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.209174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:46129)
0620 19:02:52.209290 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.209335 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 19:02:52.209897 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.209709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:57625)
0620 19:02:52.209805 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.209846 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:02:52.210065 18210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:02:52.889448 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:52.889019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:41149)
0620 19:02:52.889261 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:52.889312 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260620 19:02:53.027619 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.027172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:39251)
0620 19:02:53.027483 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:53.027539 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":69,"threads_started":1}
I20260620 19:02:53.163220 18216 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:53.163467 18216 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:53.164750 18216 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
Jun 20 19:02:53 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.130@KRBTEST.COM
Jun 20 19:02:53 dist-test-slave-tsm4 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.131@KRBTEST.COM
I20260620 19:02:53.177968 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.164912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44027 (local address 127.2.208.129:35179)
0620 19:02:53.165015 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:53.165032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:53.165136 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:53.167200 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:53.167202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:53.167571 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:53.168049 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:53.168064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.169320 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.169323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:53.170213 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:53.170221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.170913 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.170917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:53.171076 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:53.171766 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:53.171782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:53.173259 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:53.175695 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:53.175702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:53.175715 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:53.175966 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:53.176309 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:53.176316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:53.176453 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:53.177058 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:53.177064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:53.177289 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 19:02:53.177479 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 19:02:53.177781 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":40,"spinlock_wait_cycles":8320}
I20260620 19:02:53.178023 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.165126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:35179 (local address 127.2.208.130:44027)
0620 19:02:53.165490 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:53.165502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:53.165535 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:02:53.166226 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:53.166231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:53.166585 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:53.166900 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:53.168224 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.169164 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.170334 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.170791 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.170969 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:53.173500 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:53.173508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:53.173524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:53.173592 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:53.175578 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176086 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:53.176091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:53.176099 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:53.176211 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176549 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:53.176553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:53.176555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:53.176941 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:53.177084 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:53.177587 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 19:02:53.177831 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":86,"threads_started":1}
I20260620 19:02:53.178272 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.165284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:53459 (local address 127.2.208.131:37105)
0620 19:02:53.165614 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 19:02:53.165624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:02:53.165653 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:02:53.166223 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:02:53.166228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:02:53.166584 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:02:53.166838 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:02:53.168230 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.169181 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.170400 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.170839 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.171009 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:53.173931 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:02:53.173940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:02:53.173958 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:02:53.173994 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:02:53.176119 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176558 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:53.176563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:53.176568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:53.176684 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176973 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:02:53.176978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:02:53.176980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:02:53.177337 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:02:53.177480 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:02:53.177791 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:02:53.178060 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":61,"threads_started":1}
I20260620 19:02:53.178642 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.164991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37105 (local address 127.2.208.129:53459)
0620 19:02:53.165270 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:53.165283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:02:53.165384 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:02:53.166947 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:02:53.166953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:02:53.167581 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:02:53.168050 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:53.168065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.169335 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.169337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:53.170221 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:02:53.170236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:02:53.170965 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:02:53.170969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:02:53.171079 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:02:53.171815 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:02:53.171827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:02:53.173717 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:02:53.176248 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:53.176253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:53.176448 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:53.176782 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:02:53.176786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:02:53.176788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:02:53.176877 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:02:53.177453 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:02:53.177457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:02:53.177548 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 19:02:53.177667 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:02:53.177852 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":136,"spinlock_wait_cycles":26368,"thread_start_us":65,"threads_started":1}
W20260620 19:02:53.180634 17831 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
W20260620 19:02:53.180789 17830 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:53.180860 17830 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:02:53.181092 18216 raft_consensus.cc:2726] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:02:53.210918 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.210554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:35029)
0620 19:02:53.210771 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:53.210820 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260620 19:02:53.890712 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:53.890238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:47767)
0620 19:02:53.890541 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:53.890616 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260620 19:02:54.028705 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:54.028309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:59959)
0620 19:02:54.028559 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:54.028612 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260620 19:02:54.212037 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:54.211631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:40873)
0620 19:02:54.211883 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:54.211936 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260620 19:02:54.891847 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:54.891489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:48477)
0620 19:02:54.891709 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:54.891761 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260620 19:02:55.008287 18225 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:55.008445 18225 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:55.008883 18225 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
W20260620 19:02:55.009677 17830 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
W20260620 19:02:55.009760 17831 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:55.009809 17830 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:02:55.009967 18225 raft_consensus.cc:2726] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:02:55.029744 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:55.029373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:56931)
0620 19:02:55.029623 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:55.029670 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260620 19:02:55.213102 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:55.212695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:34229)
0620 19:02:55.212966 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:55.213019 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260620 19:02:55.892887 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:55.892507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:55285)
0620 19:02:55.892737 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:55.892787 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260620 19:02:56.030792 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:56.030427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:39289)
0620 19:02:56.030651 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:56.030703 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 19:02:56.214295 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:56.213919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:44583)
0620 19:02:56.214156 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:56.214209 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260620 19:02:56.866567 18231 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:02:56.866726 18231 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:02:56.867172 18231 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
W20260620 19:02:56.867913 17831 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
W20260620 19:02:56.868012 17830 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
I20260620 19:02:56.868119 17830 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:02:56.868261 18231 raft_consensus.cc:2726] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:02:56.894146 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:56.893732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:41373)
0620 19:02:56.894003 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:56.894057 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260620 19:02:57.031845 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:57.031487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:50555)
0620 19:02:57.031714 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:57.031765 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260620 19:02:57.215551 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:57.215119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:46627)
0620 19:02:57.215399 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:57.215464 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260620 19:02:57.895443 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:57.894922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:47537)
0620 19:02:57.895232 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:57.895285 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":145,"threads_started":1}
I20260620 19:02:58.032881 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:58.032513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:60163)
0620 19:02:58.032748 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:58.032800 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260620 19:02:58.216553 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:58.216193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:46493)
0620 19:02:58.216424 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:58.216474 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260620 19:02:58.896560 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:58.896176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:38823)
0620 19:02:58.896412 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:58.896466 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260620 19:02:59.034132 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:59.033698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:57833)
0620 19:02:59.033984 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:59.034037 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260620 19:02:59.217656 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:59.217237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:46845)
0620 19:02:59.217517 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:59.217570 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260620 19:02:59.897647 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:02:59.897268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:48219)
0620 19:02:59.897504 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 19:02:59.897556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260620 19:03:00.035291 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:00.034896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:44167)
0620 19:03:00.035143 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:00.035201 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260620 19:03:00.097708 18243 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:00.097841 18243 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:03:00.098342 18243 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
W20260620 19:03:00.099195 17831 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
W20260620 19:03:00.099323 17830 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
I20260620 19:03:00.099465 17830 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:03:00.099634 18243 raft_consensus.cc:2726] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:03:00.218760 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:00.218372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:42107)
0620 19:03:00.218625 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:00.218676 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260620 19:03:00.898743 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:00.898344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:60213)
0620 19:03:00.898588 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:00.898640 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260620 19:03:01.036439 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:01.036024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:39485)
0620 19:03:01.036295 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:01.036351 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260620 19:03:01.219967 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:01.219570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:53169)
0620 19:03:01.219791 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:01.219872 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260620 19:03:01.899967 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:01.899583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:49941)
0620 19:03:01.899823 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:01.899876 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260620 19:03:02.037829 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:02.037332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:47963)
0620 19:03:02.037632 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:02.037700 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260620 19:03:02.221041 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:02.220683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:48891)
0620 19:03:02.220902 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:02.220951 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260620 19:03:02.901187 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:02.900784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:48677)
0620 19:03:02.901041 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:02.901092 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260620 19:03:03.039083 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:03.038614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:55865)
0620 19:03:03.038905 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:03.038962 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260620 19:03:03.222119 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:03.221707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:54993)
0620 19:03:03.221979 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:03.222028 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260620 19:03:03.902340 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:03.901940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:50391)
0620 19:03:03.902193 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:03.902247 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260620 19:03:04.040254 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:04.039866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:54915)
0620 19:03:04.040110 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:04.040163 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260620 19:03:04.223337 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:04.222909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:39445)
0620 19:03:04.223182 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:04.223242 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260620 19:03:04.568871 18257 raft_consensus.cc:482] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:04.569057 18257 raft_consensus.cc:504] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412b00997cfa46ec81b6d76b036da116" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44027 } } peers { permanent_uuid: "1b02d027c25944eab6fac83ba4bb67a3" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37105 } } peers { permanent_uuid: "3c41b98d7c08499d9842c75a94f50129" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34857 } }
I20260620 19:03:04.569516 18257 leader_election.cc:288] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027), 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105)
W20260620 19:03:04.570286 17831 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412b00997cfa46ec81b6d76b036da116 (127.2.208.130:44027): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
W20260620 19:03:04.570416 17830 leader_election.cc:341] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b02d027c25944eab6fac83ba4bb67a3 (127.2.208.131:37105): Not found: Tablet not found: eb2e18540708432f8ff32926bbe9624a
I20260620 19:03:04.570514 17830 leader_election.cc:302] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c41b98d7c08499d9842c75a94f50129; no voters: 1b02d027c25944eab6fac83ba4bb67a3, 412b00997cfa46ec81b6d76b036da116
I20260620 19:03:04.570674 18257 raft_consensus.cc:2726] T eb2e18540708432f8ff32926bbe9624a P 3c41b98d7c08499d9842c75a94f50129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:03:04.903658 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:04.903150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:59861)
0620 19:03:04.903507 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:04.903560 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":83,"threads_started":1}
I20260620 19:03:05.041409 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:05.040996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:34837)
0620 19:03:05.041254 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:05.041312 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260620 19:03:05.224745 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:05.224287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:33521)
0620 19:03:05.224588 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:05.224646 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260620 19:03:05.904871 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:05.904440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:45279)
0620 19:03:05.904677 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:05.904756 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260620 19:03:06.042562 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:06.042124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:52983)
0620 19:03:06.042411 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:06.042469 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
I20260620 19:03:06.225967 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:06.225583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.131:37737)
0620 19:03:06.225812 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:06.225864 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260620 19:03:06.906206 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:06.905738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.129:58847)
0620 19:03:06.906063 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:06.906116 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":67,"threads_started":1}
I20260620 19:03:07.044015 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:07.043578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41125 (local address 127.2.208.130:59689)
0620 19:03:07.043853 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:07.043901 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260620 19:03:07.214962 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17779
I20260620 19:03:07.220728 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 17946
I20260620 19:03:07.225924 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18081
2026-06-20T19:03:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19481 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:03:07 dist-test-slave-tsm4 krb5kdc[18268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:03:07 dist-test-slave-tsm4 krb5kdc[18268](info): set up 2 sockets
Jun 20 19:03:07 dist-test-slave-tsm4 krb5kdc[18268](info): commencing operation
krb5kdc: starting...
W20260620 19:03:08.259570 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:03:08.379457 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:08.488472 18281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:08.488747 18281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:08.488816 18281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:08.492739 18281 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:03:08.492817 18281 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:03:08.492844 18281 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:03:08.492874 18281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:03:08.492915 18281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:08.492951 18281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:08.493003 18281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:03:08.493041 18281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:03:08.498170 18281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:08.499209 18281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:08.500222 18281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:03:08.508616 18281 server_base.cc:1034] running on GCE node
W20260620 19:03:08.508625 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.508805 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.508635 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:08.509403 18281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:08.510476 18281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:08.511631 18281 hybrid_clock.cc:648] HybridClock initialized: now 1781982188511613 us; error 30 us; skew 500 ppm
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:08.514281 18281 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:03:08.515216 18281 webserver.cc:460] Webserver started at http://127.2.208.190:46679/ using document root <none> and password file <none>
I20260620 19:03:08.515805 18281 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:08.515862 18281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:08.516024 18281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:03:08.517975 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "2a0bcf083633438087b79cd07bd13994"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "183f6c4d6e7ce311baac24d1713a6f60"
server_key_iv: "863f287faeaf7d6d6e9ce80b1264bb2b"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.518464 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "2a0bcf083633438087b79cd07bd13994"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "183f6c4d6e7ce311baac24d1713a6f60"
server_key_iv: "863f287faeaf7d6d6e9ce80b1264bb2b"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.521973 18281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:03:08.524241 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:08.525485 18281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:03:08.525614 18281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2a0bcf083633438087b79cd07bd13994"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "183f6c4d6e7ce311baac24d1713a6f60"
server_key_iv: "863f287faeaf7d6d6e9ce80b1264bb2b"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.525718 18281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:08.550000 18281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:08.552860 18281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:08.553059 18281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:08.560158 18281 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:36293
I20260620 19:03:08.560444 18348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:36293 every 8 connection(s)
I20260620 19:03:08.561103 18281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:03:08.563828 18349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:08.565717 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18281
I20260620 19:03:08.565851 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:03:08.566183 2882 external_mini_cluster.cc:1428] Setting key 321546674456c93b90860efb5b10454a
I20260620 19:03:08.570814 18349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap starting.
I20260620 19:03:08.573447 18349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:08.574156 18349 log.cc:824] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:08.576117 18349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: No bootstrap required, opened a new log
I20260620 19:03:08.580663 18349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:08.580910 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:08.567694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.2.208.190:36293)
0620 19:03:08.568109 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:08.568131 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:08.568171 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:03:08.568937 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:08.568960 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:08.569370 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:08.569735 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:08.570371 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.571772 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.572612 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.573273 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.573440 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:08.576144 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:08.576151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:08.576169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:08.576197 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:08.577985 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:08.578388 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:08.578395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:08.578401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:08.578571 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:08.578820 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:08.578823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:08.578825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:08.579303 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:08.579530 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:08.579710 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:03:08.580003 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260620 19:03:08.580909 18349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:08.581038 18349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Initialized, Role: FOLLOWER
I20260620 19:03:08.581432 18349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:08.581583 18349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:03:08.581665 18349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:03:08.581777 18349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:08.582813 18349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:08.583158 18349 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0bcf083633438087b79cd07bd13994; no voters:
I20260620 19:03:08.583485 18349 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:03:08.583596 18354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:03:08.583840 18354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Running, Role: LEADER
I20260620 19:03:08.584164 18354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:08.584743 18349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:03:08.586491 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0bcf083633438087b79cd07bd13994. Latest consensus state: current_term: 1 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:08.586549 18355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:08.586736 18355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:08.586961 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:08.587164 18364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:03:08.589752 18364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:03:08.594956 18364 catalog_manager.cc:1347] Generated new cluster ID: 535ac30e53884eb99b45c7fcf87b29d5
I20260620 19:03:08.595016 18364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:03:08.615950 18364 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:03:08.617206 18364 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:03:08.625069 18364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Generated new TSK 0
I20260620 19:03:08.625799 18364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:03:08.654538 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:08.765389 18375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:08.765638 18375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:08.765718 18375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:08.769888 18375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:08.769939 18375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:08.770045 18375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:03:08.775120 18375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:08.776191 18375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:08.777194 18375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:08.786566 18383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.786605 18381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.786602 18380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:08.786798 18375 server_base.cc:1034] running on GCE node
I20260620 19:03:08.787326 18375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:08.788048 18375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:08.789204 18375 hybrid_clock.cc:648] HybridClock initialized: now 1781982188789191 us; error 25 us; skew 500 ppm
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:08.791803 18375 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:03:08.792608 18375 webserver.cc:460] Webserver started at http://127.2.208.129:33829/ using document root <none> and password file <none>
I20260620 19:03:08.793131 18375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:08.793222 18375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:08.793469 18375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:03:08.795464 18375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "73a9182a8a374ee99d8d33255d5e46b2"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "caadbfddba8b62b3a91869ff0d73ba4f"
server_key_iv: "33445ff086baee3abb0b19086efddf77"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.796005 18375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "73a9182a8a374ee99d8d33255d5e46b2"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "caadbfddba8b62b3a91869ff0d73ba4f"
server_key_iv: "33445ff086baee3abb0b19086efddf77"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.799552 18375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 19:03:08.801735 18390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:08.802879 18375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:08.803006 18375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "73a9182a8a374ee99d8d33255d5e46b2"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "caadbfddba8b62b3a91869ff0d73ba4f"
server_key_iv: "33445ff086baee3abb0b19086efddf77"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.803113 18375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:08.816319 18375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:08.818907 18375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:08.819096 18375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:08.819715 18375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:08.820492 18375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:08.820560 18375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:08.820622 18375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:08.820668 18375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:08.829260 18375 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:38393
I20260620 19:03:08.829291 18503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:38393 every 8 connection(s)
I20260620 19:03:08.830174 18375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:03:08.830371 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18375
I20260620 19:03:08.830500 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:03:08.830801 2882 external_mini_cluster.cc:1428] Setting key e08795f790a14899833243d527599065
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:03:08.844149 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:08.831967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:42707 (local address 127.2.208.190:36293)
0620 19:03:08.832118 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:08.832123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:08.833109 (+ 986us) server_negotiation.cc:407] Connection header received
0620 19:03:08.834374 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:08.834377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:08.834461 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:08.834567 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:08.835774 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.836310 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.837429 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.837669 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.837778 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:08.840362 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:08.840365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:08.840367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:08.840389 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:08.841867 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:08.842529 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:08.842532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:08.842534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:08.842632 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:08.842957 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:08.842963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:08.842968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:08.843235 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:08.843351 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:08.843868 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 19:03:08.843984 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:03:08.844937 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:08.832446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:42707)
0620 19:03:08.832958 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:08.833000 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:08.834174 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:08.834743 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:08.834750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:08.835111 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:08.835636 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:08.835650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.836443 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.836447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:08.837300 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:08.837309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:08.837816 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:08.837820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:08.837986 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:08.838575 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:08.838596 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:08.840215 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:08.842008 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:08.842012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:08.842034 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:08.842410 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:08.842730 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:08.842732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:08.842734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:08.842858 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:08.843335 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:08.843340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:08.843631 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 19:03:08.843827 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 19:03:08.844132 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":132,"threads_started":1}
I20260620 19:03:08.846226 18504 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:08.846475 18504 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:08.847015 18504 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:08.848690 18313 ts_manager.cc:194] Registered new tserver with Master: 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393)
I20260620 19:03:08.849783 18313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:42707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:03:08.857643 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:08.964074 18509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:08.964377 18509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:08.964452 18509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:08.968298 18509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:08.968381 18509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:08.968467 18509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:03:08.973577 18509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:08.974604 18509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:08.975620 18509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:08.985193 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.985216 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:08.985194 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:08.985513 18509 server_base.cc:1034] running on GCE node
I20260620 19:03:08.985880 18509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:08.986472 18509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:08.987624 18509 hybrid_clock.cc:648] HybridClock initialized: now 1781982188987593 us; error 44 us; skew 500 ppm
Jun 20 19:03:08 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:08.990281 18509 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:03:08.991165 18509 webserver.cc:460] Webserver started at http://127.2.208.130:45795/ using document root <none> and password file <none>
I20260620 19:03:08.991770 18509 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:08.991830 18509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:08.992071 18509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:03:08.994072 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "8112f3b03f8b4fa783dc249a378c4494"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "ac5641788342e7a0b4e0f34df99e8d2d"
server_key_iv: "b103ef70a1dd88895ede0f19ac2825f8"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.994632 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "8112f3b03f8b4fa783dc249a378c4494"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "ac5641788342e7a0b4e0f34df99e8d2d"
server_key_iv: "b103ef70a1dd88895ede0f19ac2825f8"
server_key_version: "encryptionkey@0"
I20260620 19:03:08.998196 18509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:03:09.000690 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.001924 18509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:03:09.002048 18509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "8112f3b03f8b4fa783dc249a378c4494"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "ac5641788342e7a0b4e0f34df99e8d2d"
server_key_iv: "b103ef70a1dd88895ede0f19ac2825f8"
server_key_version: "encryptionkey@0"
I20260620 19:03:09.002158 18509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:09.020146 18509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:09.023283 18509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:09.023455 18509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:09.024086 18509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:09.024866 18509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:09.024910 18509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.024973 18509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:09.025007 18509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.034121 18509 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44247
I20260620 19:03:09.034154 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44247 every 8 connection(s)
I20260620 19:03:09.034966 18509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:03:09.043288 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18509
I20260620 19:03:09.043460 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:03:09.043793 2882 external_mini_cluster.cc:1428] Setting key 867c6b52a968cd8a9ecad967d3b4a707
Jun 20 19:03:09 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982188, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:09.048384 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.036647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:56679 (local address 127.2.208.190:36293)
0620 19:03:09.036805 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.036810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.037631 (+ 821us) server_negotiation.cc:407] Connection header received
0620 19:03:09.038816 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.038818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.038863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.038938 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:09.040105 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.040625 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.041705 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.041912 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.041984 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.044498 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:09.044501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:09.044503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:09.044553 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:09.046032 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.046650 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.046653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.046654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.046725 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.047120 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.047126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.047128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.047427 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:09.047535 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.048042 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.048184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:03:09.049039 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.037042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:56679)
0620 19:03:09.037486 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.037527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.038632 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.039098 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.039104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.039519 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:09.039977 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.039989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.040726 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.040728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.041583 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.041610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.042029 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.042032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.042165 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.042666 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:09.042682 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:09.044345 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:09.046148 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:09.046156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:09.046174 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:09.046550 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:09.046812 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:09.046819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:09.046823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:09.047008 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:09.047541 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:09.047546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:09.047765 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.047926 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.048172 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260620 19:03:09.050149 18638 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:09.050405 18638 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:09.051011 18638 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:09.052168 18313 ts_manager.cc:194] Registered new tserver with Master: 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247)
I20260620 19:03:09.052793 18313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:56679
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:03:09.070650 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:09.177656 18643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:09.177888 18643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:09.177928 18643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:09.181708 18643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:09.181748 18643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:09.181800 18643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:03:09.186926 18643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:09.187927 18643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:09.188825 18643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:09.197729 18648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:09.197710 18643 server_base.cc:1034] running on GCE node
W20260620 19:03:09.197696 18649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:09.197723 18651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:09.198355 18643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:09.198889 18643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:09.200039 18643 hybrid_clock.cc:648] HybridClock initialized: now 1781982189200027 us; error 25 us; skew 500 ppm
Jun 20 19:03:09 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982189, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:09.202602 18643 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:03:09.203353 18643 webserver.cc:460] Webserver started at http://127.2.208.131:33547/ using document root <none> and password file <none>
I20260620 19:03:09.203926 18643 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:09.204010 18643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:09.204231 18643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:03:09.206388 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "e136e1d520f34bff94c5170c724628cb"
format_stamp: "Formatted at 2026-06-20 19:03:09 on dist-test-slave-tsm4"
server_key: "ae417d1f091369a7fe4e81e564ec96cf"
server_key_iv: "a9ab610c55af48ee4cf26048328703fa"
server_key_version: "encryptionkey@0"
I20260620 19:03:09.206977 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "e136e1d520f34bff94c5170c724628cb"
format_stamp: "Formatted at 2026-06-20 19:03:09 on dist-test-slave-tsm4"
server_key: "ae417d1f091369a7fe4e81e564ec96cf"
server_key_iv: "a9ab610c55af48ee4cf26048328703fa"
server_key_version: "encryptionkey@0"
I20260620 19:03:09.210517 18643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:03:09.212759 18658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.213825 18643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:03:09.213968 18643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e136e1d520f34bff94c5170c724628cb"
format_stamp: "Formatted at 2026-06-20 19:03:09 on dist-test-slave-tsm4"
server_key: "ae417d1f091369a7fe4e81e564ec96cf"
server_key_iv: "a9ab610c55af48ee4cf26048328703fa"
server_key_version: "encryptionkey@0"
I20260620 19:03:09.214089 18643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:09.230732 18643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:09.233829 18643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:09.234023 18643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:09.234591 18643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:09.235334 18643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:09.235419 18643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.235486 18643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:09.235538 18643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:09.244180 18643 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:38727
I20260620 19:03:09.244205 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:38727 every 8 connection(s)
I20260620 19:03:09.245034 18643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:03:09.246197 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18643
I20260620 19:03:09.246330 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:03:09.246640 2882 external_mini_cluster.cc:1428] Setting key 846b57352339438dd464abcf4ec6bce5
Jun 20 19:03:09 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982189, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:09.257745 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.246841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39597 (local address 127.2.208.190:36293)
0620 19:03:09.246978 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.246983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.248049 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 19:03:09.249357 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.249359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.249401 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.249466 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:09.250498 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.251002 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.252053 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.252275 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.252349 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.254557 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:09.254560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:09.254562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:09.254584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:09.256095 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.256552 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.256554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.256556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.256624 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.256884 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.256886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.256887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.257050 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:09.257121 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.257531 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.257632 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:03:09.258476 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.247269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:39597)
0620 19:03:09.247868 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.247944 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.249185 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.249591 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.249597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.249938 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:09.250388 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.250400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.251101 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.251103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.251957 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.251966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.252392 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.252395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.252530 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.253008 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:09.253023 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:09.254420 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:09.256187 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:09.256191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:09.256200 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:09.256469 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:09.256695 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:09.256697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:09.256699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:09.256812 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:09.257129 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:09.257134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:09.257330 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.257489 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.257726 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":144,"threads_started":1}
I20260620 19:03:09.259569 18772 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:09.259793 18772 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:09.260219 18772 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:09.261109 18312 ts_manager.cc:194] Registered new tserver with Master: e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
I20260620 19:03:09.261646 18312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:39597
I20260620 19:03:09.272643 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:03:09.281659 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.274329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.2.208.190:36293)
0620 19:03:09.274488 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.274492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.274673 (+ 181us) server_negotiation.cc:407] Connection header received
0620 19:03:09.274814 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.274817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.274864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.274975 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:09.275678 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.276224 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.277068 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.277291 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.277403 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.278711 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:09.278714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:09.278717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:09.278740 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:09.279979 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.280440 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.280445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.280447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.280527 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:09.280797 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:09.280800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:09.280802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:09.281013 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:09.281155 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.281395 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.281501 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 19:03:09.284251 18312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:03:09.287484 18312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:03:09.300804 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.295072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.2.208.129:38393)
0620 19:03:09.295895 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.295908 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.295932 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:03:09.296016 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.296021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.296307 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.296802 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:09.296972 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.298168 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.299398 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300111 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.300238 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.300396 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.300559 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.300647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":127,"threads_started":1}
I20260620 19:03:09.300791 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.294935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38393 (local address 127.0.0.1:58640)
0620 19:03:09.295426 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.295476 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.295604 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.296472 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.296478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.296500 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:09.296649 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.296660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.298352 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.298355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.299244 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.299256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300237 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.300241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.300352 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.300399 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.300532 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.300646 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260620 19:03:09.301271 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.295691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44247 (local address 127.0.0.1:49472)
0620 19:03:09.295987 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.296013 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.296098 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.296697 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.296703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.296721 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:09.296801 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.296806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.298299 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.298302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.299705 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.299715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300801 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.300807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.300927 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.300977 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.301043 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.301096 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":140,"thread_start_us":85,"threads_started":1}
I20260620 19:03:09.301302 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.295000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38727 (local address 127.0.0.1:41800)
0620 19:03:09.295503 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.295516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.295604 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.296843 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.296846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.296866 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:09.296963 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.296968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.298118 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.298124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.299828 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.299835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300831 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.300834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.300931 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.300973 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.301024 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.301096 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":262,"spinlock_wait_cycles":3840,"thread_start_us":99,"threads_started":1}
I20260620 19:03:09.301790 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.296078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.2.208.131:38727)
0620 19:03:09.296348 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.296359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.296379 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:03:09.296496 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.296501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.296689 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.297130 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:09.297238 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.297983 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.299948 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300717 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.301175 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.301477 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.301542 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.301629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260620 19:03:09.301965 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.295856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.2.208.130:44247)
0620 19:03:09.296254 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.296265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.296286 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:03:09.296371 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.296380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.296553 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.296938 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:09.297088 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.298168 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.299841 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.300655 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.301545 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.301696 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.301740 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.301827 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":112,"threads_started":1}
I20260620 19:03:09.304141 18572 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84c98ee0404255806d12fa048894c6 (DEFAULT_TABLE table=test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:09.304189 18706 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84c98ee0404255806d12fa048894c6 (DEFAULT_TABLE table=test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:09.304154 18438 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84c98ee0404255806d12fa048894c6 (DEFAULT_TABLE table=test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:09.305013 18572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84c98ee0404255806d12fa048894c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:09.305135 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84c98ee0404255806d12fa048894c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:09.305142 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84c98ee0404255806d12fa048894c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:09.311595 18789 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Bootstrap starting.
I20260620 19:03:09.312824 18791 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Bootstrap starting.
I20260620 19:03:09.312928 18790 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap starting.
I20260620 19:03:09.314020 18789 tablet_bootstrap.cc:654] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:09.314916 18791 tablet_bootstrap.cc:654] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:09.314916 18790 tablet_bootstrap.cc:654] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:09.315843 18789 log.cc:824] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:09.316273 18790 log.cc:824] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:09.316511 18791 log.cc:824] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:09.318408 18790 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: No bootstrap required, opened a new log
I20260620 19:03:09.318615 18790 ts_tablet_manager.cc:1397] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 19:03:09.318670 18791 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: No bootstrap required, opened a new log
I20260620 19:03:09.318835 18791 ts_tablet_manager.cc:1397] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 19:03:09.320556 18789 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: No bootstrap required, opened a new log
I20260620 19:03:09.320719 18789 ts_tablet_manager.cc:1397] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 19:03:09.322479 18790 raft_consensus.cc:348] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.322731 18790 raft_consensus.cc:374] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:09.322774 18790 raft_consensus.cc:729] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a9182a8a374ee99d8d33255d5e46b2, State: Initialized, Role: FOLLOWER
I20260620 19:03:09.322604 18791 raft_consensus.cc:348] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.322858 18791 raft_consensus.cc:374] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:09.322911 18791 raft_consensus.cc:729] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e136e1d520f34bff94c5170c724628cb, State: Initialized, Role: FOLLOWER
I20260620 19:03:09.323356 18791 consensus_queue.cc:260] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.324013 18790 consensus_queue.cc:260] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.324165 18772 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
I20260620 19:03:09.324405 18791 ts_tablet_manager.cc:1428] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 19:03:09.324281 18789 raft_consensus.cc:348] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.324514 18789 raft_consensus.cc:374] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:09.324573 18789 raft_consensus.cc:729] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8112f3b03f8b4fa783dc249a378c4494, State: Initialized, Role: FOLLOWER
I20260620 19:03:09.324994 18790 ts_tablet_manager.cc:1428] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:03:09.324934 18789 consensus_queue.cc:260] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.325263 18504 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
I20260620 19:03:09.327487 18789 ts_tablet_manager.cc:1428] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:03:09.327801 18638 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
W20260620 19:03:09.331481 18505 tablet.cc:2369] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:03:09.496057 18773 tablet.cc:2369] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:03:09.518910 18796 raft_consensus.cc:482] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:09.519155 18796 raft_consensus.cc:504] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.520325 18796 leader_election.cc:288] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727), 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247)
I20260620 19:03:09.524572 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.520694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:33299 (local address 127.2.208.131:38727)
0620 19:03:09.520827 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.520830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.520962 (+ 132us) server_negotiation.cc:407] Connection header received
0620 19:03:09.521180 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.521186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.521256 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.521348 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:09.521718 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.522311 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.523516 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.523952 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.524039 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.524079 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.524372 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.524446 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:03:09.524577 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.521129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:53801 (local address 127.2.208.130:44247)
0620 19:03:09.521264 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.521267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.521435 (+ 168us) server_negotiation.cc:407] Connection header received
0620 19:03:09.521588 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.521591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.521650 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.521737 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:09.521990 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.522502 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.523637 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.524043 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.524152 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.524204 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.524410 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.524472 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:03:09.524593 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.520987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44247 (local address 127.2.208.129:53801)
0620 19:03:09.521338 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.521351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.521477 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.521742 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.521745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.521763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:09.521868 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.521874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.522618 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.522621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.523517 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.523526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.524158 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.524161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.524281 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.524298 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.524388 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.524459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"spinlock_wait_cycles":2048,"thread_start_us":100,"threads_started":1}
I20260620 19:03:09.524601 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.520597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38727 (local address 127.2.208.129:33299)
0620 19:03:09.520859 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.520875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:09.521001 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:09.521369 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:09.521371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:09.521389 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:09.521584 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.521591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.522421 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.522424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.523286 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:09.523295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.524070 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.524076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:09.524178 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.524267 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:03:09.524336 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:03:09.524421 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":52,"spinlock_wait_cycles":20864,"thread_start_us":109,"threads_started":1}
I20260620 19:03:09.525353 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84c98ee0404255806d12fa048894c6" candidate_uuid: "73a9182a8a374ee99d8d33255d5e46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8112f3b03f8b4fa783dc249a378c4494" is_pre_election: true
I20260620 19:03:09.525607 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84c98ee0404255806d12fa048894c6" candidate_uuid: "73a9182a8a374ee99d8d33255d5e46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e136e1d520f34bff94c5170c724628cb" is_pre_election: true
I20260620 19:03:09.525657 18592 raft_consensus.cc:2454] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a9182a8a374ee99d8d33255d5e46b2 in term 0.
I20260620 19:03:09.525867 18726 raft_consensus.cc:2454] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73a9182a8a374ee99d8d33255d5e46b2 in term 0.
I20260620 19:03:09.526100 18392 leader_election.cc:302] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2, 8112f3b03f8b4fa783dc249a378c4494; no voters:
I20260620 19:03:09.526326 18796 raft_consensus.cc:2781] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:03:09.526422 18796 raft_consensus.cc:482] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:03:09.526480 18796 raft_consensus.cc:3037] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:09.527557 18796 raft_consensus.cc:504] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.527884 18796 leader_election.cc:288] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 1 election: Requested vote from peers e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727), 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247)
I20260620 19:03:09.528219 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84c98ee0404255806d12fa048894c6" candidate_uuid: "73a9182a8a374ee99d8d33255d5e46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e136e1d520f34bff94c5170c724628cb"
I20260620 19:03:09.528332 18726 raft_consensus.cc:3037] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:09.528331 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84c98ee0404255806d12fa048894c6" candidate_uuid: "73a9182a8a374ee99d8d33255d5e46b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8112f3b03f8b4fa783dc249a378c4494"
I20260620 19:03:09.528434 18592 raft_consensus.cc:3037] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:09.529414 18726 raft_consensus.cc:2454] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a9182a8a374ee99d8d33255d5e46b2 in term 1.
I20260620 19:03:09.529639 18592 raft_consensus.cc:2454] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73a9182a8a374ee99d8d33255d5e46b2 in term 1.
I20260620 19:03:09.529712 18392 leader_election.cc:302] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2, e136e1d520f34bff94c5170c724628cb; no voters:
I20260620 19:03:09.529874 18796 raft_consensus.cc:2781] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:03:09.530102 18796 raft_consensus.cc:686] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 LEADER]: Becoming Leader. State: Replica: 73a9182a8a374ee99d8d33255d5e46b2, State: Running, Role: LEADER
I20260620 19:03:09.530452 18796 consensus_queue.cc:237] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:09.533432 18311 catalog_manager.cc:5495] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129). New cstate: current_term: 1 leader_uuid: "73a9182a8a374ee99d8d33255d5e46b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } health_report { overall_health: UNKNOWN } } }
W20260620 19:03:09.536041 18639 tablet.cc:2369] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:03:09.597265 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.593812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.2.208.129:38393)
0620 19:03:09.593962 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:09.593969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:09.593989 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:03:09.594060 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:09.594062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:09.594113 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:09.594220 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:03:09.594476 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.594971 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.595879 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:09.596093 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:09.596198 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:09.596484 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:03:09.596872 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:03:09.597009 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:09.597068 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:03:09.597122 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:03:09.603188 18726 raft_consensus.cc:1261] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Refusing update from remote peer 73a9182a8a374ee99d8d33255d5e46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:03:09.603188 18592 raft_consensus.cc:1261] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Refusing update from remote peer 73a9182a8a374ee99d8d33255d5e46b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:03:09.603932 18796 consensus_queue.cc:1035] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:03:09.604184 18800 consensus_queue.cc:1035] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:03:09.613025 18805 mvcc.cc:205] Tried to move back new op lower bound from 7298999048607666176 to 7298999048319160320. Current Snapshot: MvccSnapshot[applied={T|T < 7298999048607666176 or (T in {7298999048607666176})}]
I20260620 19:03:09.613178 18804 mvcc.cc:205] Tried to move back new op lower bound from 7298999048607666176 to 7298999048319160320. Current Snapshot: MvccSnapshot[applied={T|T < 7298999048607666176}]
I20260620 19:03:09.618762 18311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:03:09.618894 18311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:03:09.621001 18311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Sending DeleteTablet for 3 replicas of tablet 0d84c98ee0404255806d12fa048894c6
I20260620 19:03:09.621735 18438 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d84c98ee0404255806d12fa048894c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:09 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58640
I20260620 19:03:09.621742 18706 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d84c98ee0404255806d12fa048894c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:09 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:41800
I20260620 19:03:09.621814 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18375
I20260620 19:03:09.622490 18812 tablet_replica.cc:330] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: stopping tablet replica
I20260620 19:03:09.622807 18812 raft_consensus.cc:2229] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:03:09.623113 18812 raft_consensus.cc:2258] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:03:09.623769 18572 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d84c98ee0404255806d12fa048894c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:09 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:49472
I20260620 19:03:09.624197 18813 tablet_replica.cc:330] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: stopping tablet replica
I20260620 19:03:09.624450 18813 raft_consensus.cc:2229] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:03:09.624511 18812 ts_tablet_manager.cc:1905] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:03:09.624717 18813 raft_consensus.cc:2258] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:03:09.625829 18813 ts_tablet_manager.cc:1905] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:03:09.627874 18812 ts_tablet_manager.cc:1918] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:03:09.627977 18812 log.cc:1196] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/0d84c98ee0404255806d12fa048894c6
I20260620 19:03:09.627997 18813 ts_tablet_manager.cc:1918] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:03:09.628093 18813 log.cc:1196] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/0d84c98ee0404255806d12fa048894c6
I20260620 19:03:09.628317 18812 ts_tablet_manager.cc:1939] T 0d84c98ee0404255806d12fa048894c6 P e136e1d520f34bff94c5170c724628cb: Deleting consensus metadata
I20260620 19:03:09.628420 18813 ts_tablet_manager.cc:1939] T 0d84c98ee0404255806d12fa048894c6 P 8112f3b03f8b4fa783dc249a378c4494: Deleting consensus metadata
I20260620 19:03:09.629627 18298 catalog_manager.cc:4841] TS e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): tablet 0d84c98ee0404255806d12fa048894c6 (table test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]) successfully deleted
I20260620 19:03:09.629832 18298 catalog_manager.cc:4841] TS 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): tablet 0d84c98ee0404255806d12fa048894c6 (table test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]) successfully deleted
W20260620 19:03:09.631309 18300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:38393 (error 108)
I20260620 19:03:09.631572 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18509
I20260620 19:03:09.633306 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:09.632945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38393 (local address 127.0.0.1:58652)
0620 19:03:09.633110 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:09.633204 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 19:03:09.633443 18300 catalog_manager.cc:4584] TS 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d84c98ee0404255806d12fa048894c6: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38393: connect: Connection refused (error 111)
I20260620 19:03:09.637064 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18643
I20260620 19:03:09.642268 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18281
I20260620 19:03:29.648764 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--webserver_interface=127.2.208.190
--webserver_port=46679
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:29.755375 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:29.755597 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:29.755633 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:29.759255 18815 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:03:29.759292 18815 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:03:29.759306 18815 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:03:29.759318 18815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:03:29.759334 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:29.759351 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:29.759439 18815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:03:29.759464 18815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:03:29.764304 18815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=46679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:29.765272 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:29.766201 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:29.778261 18822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:29.778261 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:29.778314 18815 server_base.cc:1034] running on GCE node
W20260620 19:03:29.778321 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:29.779054 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:29.780118 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:29.781271 18815 hybrid_clock.cc:648] HybridClock initialized: now 1781982209781256 us; error 27 us; skew 500 ppm
Jun 20 19:03:29 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:29.786504 18815 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:03:29.787338 18815 webserver.cc:460] Webserver started at http://127.2.208.190:46679/ using document root <none> and password file <none>
I20260620 19:03:29.787943 18815 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:29.788044 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:29.791600 18815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 19:03:29.793566 18831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:29.794680 18815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:29.794798 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2a0bcf083633438087b79cd07bd13994"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "183f6c4d6e7ce311baac24d1713a6f60"
server_key_iv: "863f287faeaf7d6d6e9ce80b1264bb2b"
server_key_version: "encryptionkey@0"
I20260620 19:03:29.795318 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:29.821069 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:29.823777 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:29.823958 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:29.830528 18815 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:36293
I20260620 19:03:29.830799 18883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:36293 every 8 connection(s)
I20260620 19:03:29.831612 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:03:29.834714 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18815
I20260620 19:03:29.835124 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:38393
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=33829
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:03:29.835876 18884 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:03:29.837447 18884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap starting.
I20260620 19:03:29.854087 18884 log.cc:824] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:29.862476 18884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:03:29.862845 18884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap complete.
I20260620 19:03:29.866688 18884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:29.867069 18884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Initialized, Role: FOLLOWER
I20260620 19:03:29.867578 18884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:29.867712 18884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:03:29.867794 18884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:03:29.867908 18884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:03:29.869681 18884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:29.869993 18884 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0bcf083633438087b79cd07bd13994; no voters:
I20260620 19:03:29.870268 18884 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:03:29.870400 18888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:03:29.870653 18888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 LEADER]: Becoming Leader. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Running, Role: LEADER
I20260620 19:03:29.870940 18888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:29.871490 18884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:03:29.872285 18890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:29.872432 18890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:29.873097 18889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0bcf083633438087b79cd07bd13994. Latest consensus state: current_term: 2 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:29.873199 18889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:29.873602 18897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:03:29.878614 18897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:03:29.879300 18897 catalog_manager.cc:1259] Loaded cluster ID: 535ac30e53884eb99b45c7fcf87b29d5
I20260620 19:03:29.879379 18897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:03:29.881781 18897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:03:29.882488 18897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Loaded TSK: 0
I20260620 19:03:29.896757 18897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Generated new TSK 1
I20260620 19:03:29.897007 18897 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:03:29.949952 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:29.950187 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:29.950255 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:29.954051 18886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:29.954097 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:29.954147 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:03:29.958817 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:38393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=33829
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:29.959702 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:29.960582 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:29.969475 18912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:29.969506 18913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:29.969466 18915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:29.969748 18886 server_base.cc:1034] running on GCE node
I20260620 19:03:29.970233 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:29.970840 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:29.972011 18886 hybrid_clock.cc:648] HybridClock initialized: now 1781982209971980 us; error 51 us; skew 500 ppm
Jun 20 19:03:29 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:29.974975 18886 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:03:29.976014 18886 webserver.cc:460] Webserver started at http://127.2.208.129:33829/ using document root <none> and password file <none>
I20260620 19:03:29.976732 18886 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:29.976832 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:29.984086 18886 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 19:03:29.986673 18922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:29.988193 18886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:03:29.988310 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "73a9182a8a374ee99d8d33255d5e46b2"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "caadbfddba8b62b3a91869ff0d73ba4f"
server_key_iv: "33445ff086baee3abb0b19086efddf77"
server_key_version: "encryptionkey@0"
I20260620 19:03:29.988987 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:30.002342 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:30.006004 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:30.006192 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:30.006817 18886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:30.008126 18929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:03:30.012024 18886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:03:30.012104 18886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 19:03:30.012167 18886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:03:30.014101 18886 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:03:30.014150 18886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:03:30.014420 18929 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap starting.
I20260620 19:03:30.023624 18886 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:38393
I20260620 19:03:30.023973 19036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:38393 every 8 connection(s)
I20260620 19:03:30.024641 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:03:30.031262 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 18886
I20260620 19:03:30.031752 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:44247
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=45795
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:03:30.035710 18929 log.cc:824] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:03:30 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:30.042192 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.026727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:59739)
0620 19:03:30.027257 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.027311 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.028509 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.029595 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.029604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.030068 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:30.030782 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.030805 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.032271 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.032274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.033731 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.033745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.034707 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.034714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.034978 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.035693 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:30.035718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:30.037396 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:30.039409 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.039414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.039426 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.039672 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.039987 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.039992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.039994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.040121 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.040627 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:30.040632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:30.040862 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.041025 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.041326 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
I20260620 19:03:30.042181 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.026724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:59739 (local address 127.2.208.190:36293)
0620 19:03:30.027091 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.027111 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.027451 (+ 340us) server_negotiation.cc:407] Connection header received
0620 19:03:30.028889 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.028899 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.029185 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.029451 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:30.030966 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.032115 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.033868 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.034560 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.034712 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.037577 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:30.037582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:30.037594 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:30.037620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:30.039262 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.039785 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.039789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.039793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.039898 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.040213 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.040217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.040219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.040525 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:30.040649 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.041153 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.041388 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
I20260620 19:03:30.044611 19037 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:30.044821 19037 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:30.044986 18929 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:03:30.045301 19037 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:30.045353 18929 tablet_bootstrap.cc:492] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap complete.
I20260620 19:03:30.046774 18929 ts_tablet_manager.cc:1397] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20260620 19:03:30.047685 18848 ts_manager.cc:194] Registered new tserver with Master: 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393)
I20260620 19:03:30.049620 18929 raft_consensus.cc:348] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:30.050122 18929 raft_consensus.cc:729] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a9182a8a374ee99d8d33255d5e46b2, State: Initialized, Role: FOLLOWER
I20260620 19:03:30.050563 18929 consensus_queue.cc:260] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } }
I20260620 19:03:30.050912 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:59739
I20260620 19:03:30.051569 18929 ts_tablet_manager.cc:1428] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 19:03:30.053045 19037 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
I20260620 19:03:30.057786 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.050945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.2.208.129:38393)
0620 19:03:30.053275 (+ 2330us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.053288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.053312 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:03:30.053560 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.053568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.053743 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.053941 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:30.054206 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.055061 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.056634 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.057148 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.057267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.057391 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.057575 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.057657 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":98,"threads_started":1}
I20260620 19:03:30.057785 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.050242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38393 (local address 127.0.0.1:52620)
0620 19:03:30.051524 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.051573 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.053415 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.053934 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.053939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.053964 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:30.054097 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.054103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.055164 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.055169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.056491 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.056500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.057268 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.057271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.057374 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.057412 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.057544 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.057658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":90,"threads_started":1}
I20260620 19:03:30.058681 18971 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d84c98ee0404255806d12fa048894c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:09 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:52620
I20260620 19:03:30.059473 19046 tablet_replica.cc:330] stopping tablet replica
I20260620 19:03:30.059737 19046 raft_consensus.cc:2229] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:03:30.059971 19046 raft_consensus.cc:2258] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:03:30.061007 19046 ts_tablet_manager.cc:1905] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:03:30.064361 19046 ts_tablet_manager.cc:1918] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:03:30.064450 19046 log.cc:1196] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal/wals/0d84c98ee0404255806d12fa048894c6
I20260620 19:03:30.064697 19046 ts_tablet_manager.cc:1939] T 0d84c98ee0404255806d12fa048894c6 P 73a9182a8a374ee99d8d33255d5e46b2: Deleting consensus metadata
I20260620 19:03:30.066241 18835 catalog_manager.cc:4841] TS 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393): tablet 0d84c98ee0404255806d12fa048894c6 (table test-table [id=19dd4216e5f64ce8b400fdef528fa2e7]) successfully deleted
W20260620 19:03:30.145776 19041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:30.145992 19041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:30.146047 19041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:30.149993 19041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:30.150040 19041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:30.150134 19041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:03:30.154989 19041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:44247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=45795
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:30.155897 19041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:30.156805 19041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:03:30.165692 19041 server_base.cc:1034] running on GCE node
W20260620 19:03:30.165727 19051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:30.165684 19054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:30.165702 19052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:30.166551 19041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:30.167093 19041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:30.168239 19041 hybrid_clock.cc:648] HybridClock initialized: now 1781982210168226 us; error 25 us; skew 500 ppm
Jun 20 19:03:30 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:30.170926 19041 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:03:30.171782 19041 webserver.cc:460] Webserver started at http://127.2.208.130:45795/ using document root <none> and password file <none>
I20260620 19:03:30.172269 19041 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:30.172335 19041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:30.176040 19041 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:03:30.178223 19061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.179922 19041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:03:30.180052 19041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "8112f3b03f8b4fa783dc249a378c4494"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "ac5641788342e7a0b4e0f34df99e8d2d"
server_key_iv: "b103ef70a1dd88895ede0f19ac2825f8"
server_key_version: "encryptionkey@0"
I20260620 19:03:30.180697 19041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:30.194896 19041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:30.198055 19041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:30.198223 19041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:30.198734 19041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:30.199566 19041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:30.199627 19041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.199687 19041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:30.199716 19041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.208058 19041 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44247
I20260620 19:03:30.208088 19174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44247 every 8 connection(s)
I20260620 19:03:30.208974 19041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:03:30 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:30.218567 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19041
I20260620 19:03:30.219126 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:38727
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=33547
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:03:30.221547 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.210559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:43801 (local address 127.2.208.190:36293)
0620 19:03:30.210710 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.210715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.211477 (+ 762us) server_negotiation.cc:407] Connection header received
0620 19:03:30.212680 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.212683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.212726 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.212797 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:30.213843 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.214338 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.215397 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.215648 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.215724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.217937 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:30.217940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:30.217942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:30.217963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:30.219599 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.220068 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.220074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.220079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.220165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.220475 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.220480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.220483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.220721 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:30.220818 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.221264 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.221384 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:03:30.222551 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.210933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:43801)
0620 19:03:30.211323 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.211376 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.212502 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.212945 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.212951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.213307 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:30.213732 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.213744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.214430 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.214432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.215243 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.215251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.215766 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.215769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.215898 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.216462 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:30.216478 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:30.217779 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:30.219704 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.219708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.219717 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.219955 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.220250 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.220252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.220253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.220366 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.220840 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:30.220845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:30.221055 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.221224 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.221506 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260620 19:03:30.223708 19175 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:30.223920 19175 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:30.224355 19175 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:30.225342 18848 ts_manager.cc:194] Registered new tserver with Master: 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247)
I20260620 19:03:30.225871 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:43801
W20260620 19:03:30.327116 19178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:30.327369 19178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:30.327473 19178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:30.331281 19178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:30.331357 19178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:30.331457 19178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:03:30.336488 19178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:38727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=33547
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:30.337415 19178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:30.338338 19178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:30.347110 19186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:30.347092 19184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:30.347281 19178 server_base.cc:1034] running on GCE node
W20260620 19:03:30.347093 19183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:30.347766 19178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:30.348374 19178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:30.349536 19178 hybrid_clock.cc:648] HybridClock initialized: now 1781982210349512 us; error 46 us; skew 500 ppm
Jun 20 19:03:30 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:30.352002 19178 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:03:30.352820 19178 webserver.cc:460] Webserver started at http://127.2.208.131:33547/ using document root <none> and password file <none>
I20260620 19:03:30.353348 19178 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:30.353440 19178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:30.357005 19178 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:03:30.358991 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.360105 19178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:30.360250 19178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e136e1d520f34bff94c5170c724628cb"
format_stamp: "Formatted at 2026-06-20 19:03:09 on dist-test-slave-tsm4"
server_key: "ae417d1f091369a7fe4e81e564ec96cf"
server_key_iv: "a9ab610c55af48ee4cf26048328703fa"
server_key_version: "encryptionkey@0"
I20260620 19:03:30.360746 19178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:30.370174 19178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:30.373208 19178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:30.373420 19178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:30.373960 19178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:30.374735 19178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:30.374801 19178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.374866 19178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:30.374913 19178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:30.383143 19178 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:38727
I20260620 19:03:30.383170 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:38727 every 8 connection(s)
I20260620 19:03:30.384073 19178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:03:30.385725 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19178
Jun 20 19:03:30 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:03:30.395442 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.386364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.2.208.190:36293)
0620 19:03:30.386586 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.386592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.386851 (+ 259us) server_negotiation.cc:407] Connection header received
0620 19:03:30.387045 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.387048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.387100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.387170 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:30.388182 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.388835 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.389985 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.390337 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.390433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.391987 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:30.391990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:30.391991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:30.392012 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:30.393634 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.394125 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.394130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.394131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.394218 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.394530 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.394534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.394536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.394764 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:30.394969 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.395102 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.395218 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260620 19:03:30.398589 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.385711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:41211 (local address 127.2.208.190:36293)
0620 19:03:30.385849 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.385853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.386832 (+ 979us) server_negotiation.cc:407] Connection header received
0620 19:03:30.388439 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.388442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.388484 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.388563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:30.389815 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.390527 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.392160 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.392472 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.392577 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.394940 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:30.394947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:30.394949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:30.394977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:30.396608 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.397126 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.397128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.397129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.397193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.397510 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.397515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.397516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.397695 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:30.397812 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.398319 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.398443 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:03:30.399443 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.386150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:41211)
0620 19:03:30.386673 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.386726 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.388272 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.388682 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.388688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.389100 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:30.389683 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.389699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.390649 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.390654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.392053 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.392068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.392608 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.392611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.392749 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.393323 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:30.393339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:30.394770 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:30.396701 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.396707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.396720 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.397039 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.397282 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:30.397286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:30.397288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:30.397419 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:30.397815 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:30.397842 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:30.398101 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.398280 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.398590 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260620 19:03:30.400382 19307 heartbeater.cc:347] Connected to a master server at 127.2.208.190:36293
I20260620 19:03:30.400612 19307 heartbeater.cc:464] Registering TS with master...
I20260620 19:03:30.401044 19307 heartbeater.cc:511] Master 127.2.208.190:36293 requested a full tablet report, sending...
I20260620 19:03:30.401942 18848 ts_manager.cc:194] Registered new tserver with Master: e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
I20260620 19:03:30.402470 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:41211
I20260620 19:03:30.408818 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:03:30.412254 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.409377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.2.208.190:36293)
0620 19:03:30.409473 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.409477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.409631 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:03:30.409747 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.409749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.409828 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.409903 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:03:30.410076 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.410518 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.411293 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.411551 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.411623 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.411890 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:03:30.412001 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 19:03:30.412177 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41504: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 19:03:30.412331 19039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41504: token expired
W20260620 19:03:30.412676 18777 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.208.190:36293)
I20260620 19:03:30.419528 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.413069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.2.208.190:36293)
0620 19:03:30.413204 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.413208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.413235 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:03:30.413334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.413337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.413374 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.413454 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:30.414052 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.414472 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.415150 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.415327 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.415470 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.416648 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:30.416651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:30.416652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:30.416667 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:30.417761 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.418172 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.418176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.418178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.418278 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:30.418580 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:30.418583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:30.418585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:30.418849 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:30.418978 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.419286 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.419418 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:03:30.421594 18848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:03:30.424659 18848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:03:30.431407 18971 tablet_service.cc:1467] Processing CreateTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf (DEFAULT_TABLE table=test-table [id=ae780e26bf6d4303ae2839c2dfde6a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:30.432041 18971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eded847b9a864ef9b5f6dde5f5357cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:30.434445 18929 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap starting.
I20260620 19:03:30.435163 18929 tablet_bootstrap.cc:654] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:30.435918 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.430797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38727 (local address 127.0.0.1:55804)
0620 19:03:30.431186 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.431199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.431277 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.431998 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.432001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.432018 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:30.432095 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.432101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.433304 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.433306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.434356 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.434367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.435427 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.435432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.435531 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.435561 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.435654 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.435745 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":59,"threads_started":1}
I20260620 19:03:30.436131 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.431154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.2.208.131:38727)
0620 19:03:30.431544 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.431557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.431589 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:03:30.431659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.431663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.431852 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.432005 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:30.432193 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.433200 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.434476 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.435288 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.435428 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.435568 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.435887 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.435980 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":78,"threads_started":1}
I20260620 19:03:30.436301 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.430368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44247 (local address 127.0.0.1:34316)
0620 19:03:30.430480 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.430494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.430571 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.431505 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.431507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.431524 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:30.431628 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.431634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.433185 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.433188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.434321 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.434331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.435944 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.435947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.436032 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.436053 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.436101 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.436178 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:03:30.436870 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.430680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.2.208.130:44247)
0620 19:03:30.430998 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.431014 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.431033 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:03:30.431130 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.431135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.431347 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.431521 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:30.431836 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.433058 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.434462 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.435820 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.436472 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.436612 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.436655 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.436732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":148,"threads_started":1}
I20260620 19:03:30.437001 18929 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: No bootstrap required, opened a new log
I20260620 19:03:30.437085 18929 ts_tablet_manager.cc:1397] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:03:30.437479 18929 raft_consensus.cc:348] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.437583 18929 raft_consensus.cc:374] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:30.437633 18929 raft_consensus.cc:729] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a9182a8a374ee99d8d33255d5e46b2, State: Initialized, Role: FOLLOWER
I20260620 19:03:30.437752 18929 consensus_queue.cc:260] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.437985 18929 ts_tablet_manager.cc:1428] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:03:30.439085 19241 tablet_service.cc:1467] Processing CreateTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf (DEFAULT_TABLE table=test-table [id=ae780e26bf6d4303ae2839c2dfde6a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:30.439199 19109 tablet_service.cc:1467] Processing CreateTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf (DEFAULT_TABLE table=test-table [id=ae780e26bf6d4303ae2839c2dfde6a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:03:30.440078 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eded847b9a864ef9b5f6dde5f5357cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:30.440086 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eded847b9a864ef9b5f6dde5f5357cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:03:30.446044 19320 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Bootstrap starting.
I20260620 19:03:30.446046 19321 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Bootstrap starting.
I20260620 19:03:30.448388 19321 tablet_bootstrap.cc:654] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:30.448438 19320 tablet_bootstrap.cc:654] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Neither blocks nor log segments found. Creating new log.
I20260620 19:03:30.449086 19320 log.cc:824] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:30.449095 19321 log.cc:824] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:30.450603 19320 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: No bootstrap required, opened a new log
I20260620 19:03:30.450611 19321 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: No bootstrap required, opened a new log
I20260620 19:03:30.450793 19320 ts_tablet_manager.cc:1397] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:03:30.450791 19321 ts_tablet_manager.cc:1397] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:03:30.453699 19320 raft_consensus.cc:348] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.453699 19321 raft_consensus.cc:348] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.453887 19320 raft_consensus.cc:374] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:30.453909 19321 raft_consensus.cc:374] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:03:30.453938 19320 raft_consensus.cc:729] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8112f3b03f8b4fa783dc249a378c4494, State: Initialized, Role: FOLLOWER
I20260620 19:03:30.453966 19321 raft_consensus.cc:729] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e136e1d520f34bff94c5170c724628cb, State: Initialized, Role: FOLLOWER
I20260620 19:03:30.454342 19321 consensus_queue.cc:260] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.454344 19320 consensus_queue.cc:260] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.454903 19175 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
I20260620 19:03:30.454903 19307 heartbeater.cc:503] Master 127.2.208.190:36293 was elected leader, sending a full tablet report...
I20260620 19:03:30.455229 19320 ts_tablet_manager.cc:1428] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 19:03:30.455809 19321 ts_tablet_manager.cc:1428] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 19:03:30.459800 19176 tablet.cc:2369] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:03:30.526815 19038 tablet.cc:2369] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:03:30.635020 19308 tablet.cc:2369] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:03:30.718328 19325 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:30.718562 19325 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.719758 19325 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727), 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393)
I20260620 19:03:30.723594 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.720115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:53391 (local address 127.2.208.131:38727)
0620 19:03:30.720253 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.720257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.720527 (+ 270us) server_negotiation.cc:407] Connection header received
0620 19:03:30.720671 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.720674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.720722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.720795 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:30.721067 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.721549 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.722609 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.722984 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.723122 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.723175 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.723409 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.723479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:03:30.723595 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.719991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38393 (local address 127.2.208.130:54609)
0620 19:03:30.720105 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.720122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.720212 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.720657 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.720661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.720679 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:30.720811 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.720818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.721596 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.721601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.722398 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.722406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.723113 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.723116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.723224 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.723266 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.723322 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.723424 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":4}
I20260620 19:03:30.723604 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.720119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:54609 (local address 127.2.208.129:38393)
0620 19:03:30.720393 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.720397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.720411 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:03:30.720474 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.720479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.720537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.720639 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:03:30.720930 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.721488 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.722541 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.722983 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.723123 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.723177 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.723383 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.723467 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260620 19:03:30.723611 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.719994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38727 (local address 127.2.208.130:53391)
0620 19:03:30.720426 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.720440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:30.720545 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:30.720814 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:30.720816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:30.720829 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:03:30.720951 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.720958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.721670 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.721673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.722484 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:30.722496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.723124 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.723127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:30.723242 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.723278 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:03:30.723393 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:03:30.723467 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":138,"thread_start_us":78,"threads_started":1}
I20260620 19:03:30.724243 18991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eded847b9a864ef9b5f6dde5f5357cdf" candidate_uuid: "8112f3b03f8b4fa783dc249a378c4494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a9182a8a374ee99d8d33255d5e46b2" is_pre_election: true
I20260620 19:03:30.724479 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eded847b9a864ef9b5f6dde5f5357cdf" candidate_uuid: "8112f3b03f8b4fa783dc249a378c4494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e136e1d520f34bff94c5170c724628cb" is_pre_election: true
I20260620 19:03:30.724571 18991 raft_consensus.cc:2454] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8112f3b03f8b4fa783dc249a378c4494 in term 0.
I20260620 19:03:30.724777 19261 raft_consensus.cc:2454] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8112f3b03f8b4fa783dc249a378c4494 in term 0.
I20260620 19:03:30.725101 19065 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2, 8112f3b03f8b4fa783dc249a378c4494; no voters:
I20260620 19:03:30.725406 19325 raft_consensus.cc:2781] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:03:30.725488 19325 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:03:30.725536 19325 raft_consensus.cc:3037] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:30.726534 19325 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.726888 19325 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [CANDIDATE]: Term 1 election: Requested vote from peers e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727), 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393)
I20260620 19:03:30.727203 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eded847b9a864ef9b5f6dde5f5357cdf" candidate_uuid: "8112f3b03f8b4fa783dc249a378c4494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e136e1d520f34bff94c5170c724628cb"
I20260620 19:03:30.727250 18991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eded847b9a864ef9b5f6dde5f5357cdf" candidate_uuid: "8112f3b03f8b4fa783dc249a378c4494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a9182a8a374ee99d8d33255d5e46b2"
I20260620 19:03:30.727346 19261 raft_consensus.cc:3037] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:30.727360 18991 raft_consensus.cc:3037] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:03:30.728381 19261 raft_consensus.cc:2454] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8112f3b03f8b4fa783dc249a378c4494 in term 1.
I20260620 19:03:30.728394 18991 raft_consensus.cc:2454] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8112f3b03f8b4fa783dc249a378c4494 in term 1.
I20260620 19:03:30.728670 19065 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2, 8112f3b03f8b4fa783dc249a378c4494; no voters:
I20260620 19:03:30.728833 19325 raft_consensus.cc:2781] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:03:30.729055 19325 raft_consensus.cc:686] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 1 LEADER]: Becoming Leader. State: Replica: 8112f3b03f8b4fa783dc249a378c4494, State: Running, Role: LEADER
I20260620 19:03:30.729359 19325 consensus_queue.cc:237] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:30.732292 18848 catalog_manager.cc:5495] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "8112f3b03f8b4fa783dc249a378c4494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } health_report { overall_health: UNKNOWN } } }
I20260620 19:03:30.796205 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.792912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.2.208.130:44247)
0620 19:03:30.793031 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:30.793034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:30.793065 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:03:30.793141 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:30.793143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:30.793216 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:30.793319 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:03:30.793552 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.794101 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.794974 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:30.795169 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:30.795242 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:30.795500 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:03:30.795894 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:03:30.795977 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:30.796043 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:03:30.796086 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:03:30.801950 18991 raft_consensus.cc:1261] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Refusing update from remote peer 8112f3b03f8b4fa783dc249a378c4494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:03:30.801954 19261 raft_consensus.cc:1261] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Refusing update from remote peer 8112f3b03f8b4fa783dc249a378c4494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:03:30.802631 19328 consensus_queue.cc:1035] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:03:30.802810 19325 consensus_queue.cc:1035] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:03:30.808485 19331 mvcc.cc:205] Tried to move back new op lower bound from 7298999135437955072 to 7298999135150096384. Current Snapshot: MvccSnapshot[applied={T|T < 7298999135437955072}]
I20260620 19:03:30.809557 19333 mvcc.cc:205] Tried to move back new op lower bound from 7298999135437955072 to 7298999135150096384. Current Snapshot: MvccSnapshot[applied={T|T < 7298999135437955072}]
I20260620 19:03:30.816951 18848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:03:30.817131 18848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:03:30.819279 18848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Sending DeleteTablet for 3 replicas of tablet eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:30.819965 18971 tablet_service.cc:1514] Processing DeleteTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:30 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:52620
I20260620 19:03:30.820015 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18886
I20260620 19:03:30.820593 19241 tablet_service.cc:1514] Processing DeleteTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:30 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:55804
I20260620 19:03:30.821051 19341 tablet_replica.cc:330] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: stopping tablet replica
I20260620 19:03:30.821379 19341 raft_consensus.cc:2229] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:03:30.821485 19109 tablet_service.cc:1514] Processing DeleteTablet for tablet eded847b9a864ef9b5f6dde5f5357cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:03:30 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:34316
I20260620 19:03:30.821686 19341 raft_consensus.cc:2258] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:03:30.821892 19342 tablet_replica.cc:330] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: stopping tablet replica
I20260620 19:03:30.822149 19342 raft_consensus.cc:2229] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:03:30.822450 19342 raft_consensus.cc:2258] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:03:30.823217 19341 ts_tablet_manager.cc:1905] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:03:30.823740 19342 ts_tablet_manager.cc:1905] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:03:30.826306 19342 ts_tablet_manager.cc:1918] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:03:30.826399 19342 log.cc:1196] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:30.826488 19341 ts_tablet_manager.cc:1918] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:03:30.826586 19341 log.cc:1196] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:30.826936 19342 ts_tablet_manager.cc:1939] T eded847b9a864ef9b5f6dde5f5357cdf P 8112f3b03f8b4fa783dc249a378c4494: Deleting consensus metadata
I20260620 19:03:30.826936 19341 ts_tablet_manager.cc:1939] T eded847b9a864ef9b5f6dde5f5357cdf P e136e1d520f34bff94c5170c724628cb: Deleting consensus metadata
I20260620 19:03:30.828050 18833 catalog_manager.cc:4841] TS 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): tablet eded847b9a864ef9b5f6dde5f5357cdf (table test-table [id=ae780e26bf6d4303ae2839c2dfde6a8f]) successfully deleted
I20260620 19:03:30.828230 18833 catalog_manager.cc:4841] TS e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): tablet eded847b9a864ef9b5f6dde5f5357cdf (table test-table [id=ae780e26bf6d4303ae2839c2dfde6a8f]) successfully deleted
W20260620 19:03:30.829161 18835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:38393 (error 108)
I20260620 19:03:30.829733 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19041
I20260620 19:03:30.830227 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:30.829851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38393 (local address 127.0.0.1:52622)
0620 19:03:30.830025 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:30.830126 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 19:03:30.830420 18835 catalog_manager.cc:4584] TS 73a9182a8a374ee99d8d33255d5e46b2 (127.2.208.129:38393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eded847b9a864ef9b5f6dde5f5357cdf: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38393: connect: Connection refused (error 111)
I20260620 19:03:30.835662 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19178
I20260620 19:03:30.841984 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 18815
I20260620 19:03:50.848518 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--webserver_interface=127.2.208.190
--webserver_port=46679
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:50.957082 19344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:50.957336 19344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:50.957412 19344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:50.961457 19344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:03:50.961501 19344 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:03:50.961516 19344 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:03:50.961530 19344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:03:50.961606 19344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:50.961628 19344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:50.961685 19344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:03:50.961711 19344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:03:50.966972 19344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=46679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:50.967979 19344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:50.969027 19344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:50.977813 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:50.977802 19350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:50.977789 19349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:50.978127 19344 server_base.cc:1034] running on GCE node
I20260620 19:03:50.978767 19344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:50.979816 19344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:50.980983 19344 hybrid_clock.cc:648] HybridClock initialized: now 1781982230980969 us; error 26 us; skew 500 ppm
Jun 20 19:03:50 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:50.988837 19344 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:03:50.989712 19344 webserver.cc:460] Webserver started at http://127.2.208.190:46679/ using document root <none> and password file <none>
I20260620 19:03:50.990226 19344 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:50.990284 19344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:50.994035 19344 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:50.996031 19359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:50.997231 19344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:50.997372 19344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2a0bcf083633438087b79cd07bd13994"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "183f6c4d6e7ce311baac24d1713a6f60"
server_key_iv: "863f287faeaf7d6d6e9ce80b1264bb2b"
server_key_version: "encryptionkey@0"
I20260620 19:03:50.997884 19344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:51.022495 19344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:51.025799 19344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:51.026010 19344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:51.033195 19344 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:36293
I20260620 19:03:51.033483 19411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:36293 every 8 connection(s)
I20260620 19:03:51.034281 19344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:03:51.035133 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19344
I20260620 19:03:51.035679 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:38393
--local_ip_for_outbound_sockets=127.2.208.129
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=33829
--webserver_interface=127.2.208.129
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:03:51.039474 19412 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:03:51.041039 19412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap starting.
I20260620 19:03:51.058220 19412 log.cc:824] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:51.068944 19412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:03:51.069312 19412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: Bootstrap complete.
I20260620 19:03:51.073439 19412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:51.073907 19412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Initialized, Role: FOLLOWER
I20260620 19:03:51.074373 19412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:51.074479 19412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:03:51.074563 19412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:03:51.074673 19412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:03:51.076498 19412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:51.076822 19412 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0bcf083633438087b79cd07bd13994; no voters:
I20260620 19:03:51.077112 19412 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:03:51.077275 19416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:03:51.077652 19416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [term 3 LEADER]: Becoming Leader. State: Replica: 2a0bcf083633438087b79cd07bd13994, State: Running, Role: LEADER
I20260620 19:03:51.078083 19416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } }
I20260620 19:03:51.078267 19412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:03:51.079461 19418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0bcf083633438087b79cd07bd13994. Latest consensus state: current_term: 3 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:51.079577 19418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:51.080260 19417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2a0bcf083633438087b79cd07bd13994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0bcf083633438087b79cd07bd13994" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 36293 } } }
I20260620 19:03:51.080590 19417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994 [sys.catalog]: This master's current role is: LEADER
I20260620 19:03:51.083081 19433 catalog_manager.cc:1259] Loaded cluster ID: 535ac30e53884eb99b45c7fcf87b29d5
I20260620 19:03:51.083137 19433 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: loading cluster ID for follower catalog manager: success
I20260620 19:03:51.085728 19433 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a0bcf083633438087b79cd07bd13994: acquiring CA information for follower catalog manager: success
F20260620 19:03:51.086903 19433 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08528a79ad google::LogMessage::Fail() at ??:0
@ 0x7f08528aa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f08528a74de google::LogMessage::Flush() at ??:0
@ 0x7f08528a8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f085556b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f085d20c989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f085d1e9023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f085d1e954b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f085d1e2652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f085d1e19e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f085d228541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0854aea512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f085612e6db start_thread at ??:0
@ 0x7f08513a561f clone at ??:0
W20260620 19:03:51.169577 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:51.169831 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:51.169880 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:51.174182 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:51.174237 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:51.174328 19414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:03:51.179661 19414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:38393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=33829
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:51.180716 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:51.181746 19414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:51.191233 19454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.191215 19453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.191213 19456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:51.191753 19414 server_base.cc:1034] running on GCE node
I20260620 19:03:51.192142 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:51.192782 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:51.193958 19414 hybrid_clock.cc:648] HybridClock initialized: now 1781982231193929 us; error 44 us; skew 500 ppm
Jun 20 19:03:51 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:51.196816 19414 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:03:51.197783 19414 webserver.cc:460] Webserver started at http://127.2.208.129:33829/ using document root <none> and password file <none>
I20260620 19:03:51.198333 19414 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:51.198423 19414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:51.202306 19414 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:51.204543 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.205897 19414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:03:51.206017 19414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "73a9182a8a374ee99d8d33255d5e46b2"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "caadbfddba8b62b3a91869ff0d73ba4f"
server_key_iv: "33445ff086baee3abb0b19086efddf77"
server_key_version: "encryptionkey@0"
I20260620 19:03:51.206557 19414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:51.223713 19414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:51.226737 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:51.226939 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:51.227577 19414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:51.228833 19470 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:03:51.234179 19414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:03:51.234275 19414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 19:03:51.234362 19414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:03:51.236064 19414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:03:51.236133 19414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 19:03:51.236407 19470 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap starting.
I20260620 19:03:51.246568 19414 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:38393
I20260620 19:03:51.246831 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:38393 every 8 connection(s)
I20260620 19:03:51.247829 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:03:51.252530 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19414
I20260620 19:03:51.253036 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:44247
--local_ip_for_outbound_sockets=127.2.208.130
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=45795
--webserver_interface=127.2.208.130
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:03:51.258023 19470 log.cc:824] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:03:51.266623 19470 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:03:51.267055 19470 tablet_bootstrap.cc:492] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Bootstrap complete.
I20260620 19:03:51.268326 19470 ts_tablet_manager.cc:1397] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.016s
I20260620 19:03:51.271327 19470 raft_consensus.cc:348] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:51.271878 19470 raft_consensus.cc:729] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a9182a8a374ee99d8d33255d5e46b2, State: Initialized, Role: FOLLOWER
I20260620 19:03:51.272378 19470 consensus_queue.cc:260] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:51.273516 19470 ts_tablet_manager.cc:1428] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 19:03:51.291916 19580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:03:51.249781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:49915)
0620 19:03:51.250244 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.250296 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:51.251107 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:51.291085 (+ 39978us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
W20260620 19:03:51.292603 19578 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:36293 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:36293: Client connection negotiation failed: client connection to 127.2.208.190:36293: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:03:51.293509 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.293237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:37081)
0620 19:03:51.293377 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.293444 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 19:03:51.294186 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.293982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:44649)
0620 19:03:51.294085 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.294129 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:03:51.294389 19578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:03:51.378959 19581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:51.379223 19581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:51.379302 19581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:51.383164 19581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:51.383234 19581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:51.383307 19581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:03:51.388335 19581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:44247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=45795
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:51.389353 19581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:51.390321 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:51.399466 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.399479 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.399489 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:51.399698 19581 server_base.cc:1034] running on GCE node
I20260620 19:03:51.400210 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:51.400791 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:51.401952 19581 hybrid_clock.cc:648] HybridClock initialized: now 1781982231401925 us; error 38 us; skew 500 ppm
Jun 20 19:03:51 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:51.404664 19581 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:03:51.405442 19581 webserver.cc:460] Webserver started at http://127.2.208.130:45795/ using document root <none> and password file <none>
I20260620 19:03:51.405953 19581 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:51.406009 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:51.409767 19581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:03:51.411880 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.413021 19581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:03:51.413108 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "8112f3b03f8b4fa783dc249a378c4494"
format_stamp: "Formatted at 2026-06-20 19:03:08 on dist-test-slave-tsm4"
server_key: "ac5641788342e7a0b4e0f34df99e8d2d"
server_key_iv: "b103ef70a1dd88895ede0f19ac2825f8"
server_key_version: "encryptionkey@0"
I20260620 19:03:51.413583 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:51.430438 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:51.433634 19581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:51.433777 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:51.434262 19581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:51.434988 19581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:51.435032 19581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.435070 19581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:51.435088 19581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.443436 19581 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:44247
I20260620 19:03:51.443511 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:44247 every 8 connection(s)
I20260620 19:03:51.444381 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:03:51.447944 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.446332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:46941)
0620 19:03:51.446792 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.447206 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
W20260620 19:03:51.448346 19712 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:36293 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:36293: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
I20260620 19:03:51.448917 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.448723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:42619)
0620 19:03:51.448824 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.448868 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:03:51.449512 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.449339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:46711)
0620 19:03:51.449424 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.449470 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:03:51.449699 19712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:03:51.452201 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19581
I20260620 19:03:51.452558 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:38727
--local_ip_for_outbound_sockets=127.2.208.131
--tserver_master_addrs=127.2.208.190:36293
--webserver_port=33547
--webserver_interface=127.2.208.131
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:03:51.559338 19715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:03:51.559590 19715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:03:51.559629 19715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:03:51.563450 19715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:03:51.563494 19715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:03:51.563548 19715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:03:51.568539 19715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:38727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=33547
--tserver_master_addrs=127.2.208.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:03:51.569397 19715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:03:51.570272 19715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:03:51.579113 19721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.579123 19723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:03:51.579126 19720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:03:51.579309 19715 server_base.cc:1034] running on GCE node
I20260620 19:03:51.579785 19715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:03:51.580370 19715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:03:51.581524 19715 hybrid_clock.cc:648] HybridClock initialized: now 1781982231581505 us; error 51 us; skew 500 ppm
Jun 20 19:03:51 dist-test-slave-tsm4 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:03:51.584167 19715 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:03:51.584960 19715 webserver.cc:460] Webserver started at http://127.2.208.131:33547/ using document root <none> and password file <none>
I20260620 19:03:51.585490 19715 fs_manager.cc:359] Metadata directory not provided
I20260620 19:03:51.585551 19715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:03:51.589141 19715 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:03:51.591063 19730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.592269 19715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:03:51.592414 19715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "e136e1d520f34bff94c5170c724628cb"
format_stamp: "Formatted at 2026-06-20 19:03:09 on dist-test-slave-tsm4"
server_key: "ae417d1f091369a7fe4e81e564ec96cf"
server_key_iv: "a9ab610c55af48ee4cf26048328703fa"
server_key_version: "encryptionkey@0"
I20260620 19:03:51.592957 19715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:03:51.606340 19715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:03:51.609210 19715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:03:51.609405 19715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:03:51.609958 19715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:03:51.610869 19715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:03:51.610929 19715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.611006 19715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:03:51.611038 19715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:03:51.619781 19715 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:38727
I20260620 19:03:51.619835 19843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:38727 every 8 connection(s)
I20260620 19:03:51.620759 19715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:03:51.624256 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.622744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:46391)
0620 19:03:51.623169 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.623606 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
W20260620 19:03:51.624676 19844 heartbeater.cc:644] Failed to heartbeat to 127.2.208.190:36293 (0 consecutive failures): Network error: Failed to ping master at 127.2.208.190:36293: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
I20260620 19:03:51.625257 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.625041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:39479)
0620 19:03:51.625148 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.625204 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 19:03:51.625891 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:51.625692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:54941)
0620 19:03:51.625794 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:51.625837 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:03:51.626068 19844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:03:51.628435 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19715
I20260620 19:03:52.295486 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.294982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:33241)
0620 19:03:52.295245 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:52.295303 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260620 19:03:52.450562 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.450207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:41287)
0620 19:03:52.450422 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:52.450470 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260620 19:03:52.626977 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.626623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:36687)
0620 19:03:52.626835 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:52.626885 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260620 19:03:52.877755 19851 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:52.877976 19851 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:52.879215 19851 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247), e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
Jun 20 19:03:52 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.131@KRBTEST.COM
Jun 20 19:03:52 dist-test-slave-tsm4 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.130@KRBTEST.COM
I20260620 19:03:52.896296 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.879465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:44247 (local address 127.2.208.129:57351)
0620 19:03:52.879690 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:52.879703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:52.879802 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:52.881638 (+ 1836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:52.881643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:52.882205 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:52.883075 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:52.883095 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.884962 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.884966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:52.886647 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:52.886659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.887642 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.887646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:52.887963 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:52.888706 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:52.888734 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:52.890960 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:52.893282 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:52.893288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:52.893303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:52.893790 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:52.894176 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:52.894179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:52.894181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:52.894407 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:52.895134 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:52.895140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:52.895514 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 19:03:52.895745 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 19:03:52.896082 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":454,"spinlock_wait_cycles":49664,"thread_start_us":92,"threads_started":1}
I20260620 19:03:52.896342 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.879577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:57351 (local address 127.2.208.130:44247)
0620 19:03:52.879855 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:52.879882 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:52.879926 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:03:52.880804 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:52.880812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:52.881257 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:52.881599 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:52.883259 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.884827 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.886773 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.887519 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.887686 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:52.891137 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:52.891145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:52.891160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:52.891193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:52.893153 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:52.893908 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:52.893914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:52.893920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:52.894065 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:52.894510 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:52.894513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:52.894515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:52.895004 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:52.895129 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:52.895854 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 19:03:52.896163 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
I20260620 19:03:52.896534 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.880081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:44823 (local address 127.2.208.131:38727)
0620 19:03:52.880384 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 19:03:52.880404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:03:52.880443 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:03:52.881219 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:03:52.881226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:03:52.881580 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:03:52.881858 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:03:52.883258 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.884810 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.886773 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.887519 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.887686 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:52.890986 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:03:52.890991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:03:52.891003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:03:52.891051 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:03:52.893153 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:52.893908 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:52.893914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:52.893920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:52.894065 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:03:52.894510 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:03:52.894513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:03:52.894515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:03:52.895006 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:03:52.895129 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:03:52.896068 (+ 939us) server_negotiation.cc:299] Negotiation successful
0620 19:03:52.896327 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260620 19:03:52.896332 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:52.879818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:38727 (local address 127.2.208.129:44823)
0620 19:03:52.880117 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:52.880131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:03:52.880219 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:03:52.881811 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:03:52.881814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:03:52.882205 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:03:52.883075 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:52.883095 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.884962 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.884966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:52.886648 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:03:52.886659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:03:52.887642 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:03:52.887646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:03:52.887966 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:03:52.888732 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:03:52.888749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:03:52.890765 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:03:52.893282 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:52.893288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:52.893303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:52.893788 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:52.894176 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:03:52.894179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:03:52.894181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:03:52.894407 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:03:52.895134 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:03:52.895140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:03:52.895514 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 19:03:52.895745 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 19:03:52.896165 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":608,"spinlock_wait_cycles":15232,"thread_start_us":51,"threads_started":1}
W20260620 19:03:52.899544 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
W20260620 19:03:52.899752 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:52.899837 19465 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2; no voters: 8112f3b03f8b4fa783dc249a378c4494, e136e1d520f34bff94c5170c724628cb
I20260620 19:03:52.900112 19851 raft_consensus.cc:2726] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:03:53.296581 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:53.296309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:37029)
0620 19:03:53.296435 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:53.296491 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 19:03:53.451747 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:53.451307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:48633)
0620 19:03:53.451602 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:53.451665 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260620 19:03:53.628043 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:53.627685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:43969)
0620 19:03:53.627910 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:53.627967 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260620 19:03:54.297758 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:54.297384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:49725)
0620 19:03:54.297613 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:54.297663 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260620 19:03:54.452874 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:54.452411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:51825)
0620 19:03:54.452655 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:54.452714 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260620 19:03:54.549945 19860 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:54.550107 19860 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:54.550529 19860 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247), e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
W20260620 19:03:54.551338 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
W20260620 19:03:54.551496 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:54.551538 19465 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2; no voters: 8112f3b03f8b4fa783dc249a378c4494, e136e1d520f34bff94c5170c724628cb
I20260620 19:03:54.551658 19860 raft_consensus.cc:2726] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:03:54.629261 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:54.628783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:52043)
0620 19:03:54.629028 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:54.629144 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260620 19:03:55.298789 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:55.298409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:38717)
0620 19:03:55.298650 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:55.298699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260620 19:03:55.453989 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:55.453614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:55183)
0620 19:03:55.453849 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:55.453903 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260620 19:03:55.630446 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:55.630053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:40439)
0620 19:03:55.630303 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:55.630359 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260620 19:03:56.300151 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:56.299694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:47317)
0620 19:03:56.299980 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:56.300047 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
I20260620 19:03:56.455116 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:56.454719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:59785)
0620 19:03:56.454968 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:56.455021 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260620 19:03:56.631618 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:56.631168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:35667)
0620 19:03:56.631465 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:56.631519 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260620 19:03:57.301178 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:57.300813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:40825)
0620 19:03:57.301033 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:57.301083 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260620 19:03:57.456127 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:57.455726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:35169)
0620 19:03:57.455981 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:57.456031 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260620 19:03:57.632638 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:57.632259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:41527)
0620 19:03:57.632497 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:57.632550 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260620 19:03:57.825469 19871 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:03:57.825618 19871 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:03:57.826009 19871 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247), e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
W20260620 19:03:57.826762 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
W20260620 19:03:57.826916 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:03:57.826970 19465 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2; no voters: 8112f3b03f8b4fa783dc249a378c4494, e136e1d520f34bff94c5170c724628cb
I20260620 19:03:57.827140 19871 raft_consensus.cc:2726] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:03:58.302335 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:58.301959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:47663)
0620 19:03:58.302193 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:58.302241 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260620 19:03:58.457217 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:58.456810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:50735)
0620 19:03:58.457072 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:58.457122 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260620 19:03:58.633827 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:58.633409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:50203)
0620 19:03:58.633681 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:58.633733 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":67,"threads_started":1}
I20260620 19:03:59.303552 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:59.303090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:59363)
0620 19:03:59.303340 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:59.303452 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260620 19:03:59.458359 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:59.457980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:52397)
0620 19:03:59.458215 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:59.458270 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260620 19:03:59.634898 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:03:59.634535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:52305)
0620 19:03:59.634761 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 19:03:59.634812 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
I20260620 19:04:00.304720 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:00.304308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:47521)
0620 19:04:00.304560 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:00.304618 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260620 19:04:00.459543 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:00.459105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:51997)
0620 19:04:00.459343 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:00.459450 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260620 19:04:00.636174 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:00.635760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:56019)
0620 19:04:00.636022 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:00.636087 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260620 19:04:01.305909 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:01.305484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:52463)
0620 19:04:01.305737 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:01.305808 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260620 19:04:01.460668 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:01.460227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:35941)
0620 19:04:01.460492 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:01.460557 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260620 19:04:01.637238 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:01.636860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:50457)
0620 19:04:01.637090 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:01.637152 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260620 19:04:01.956840 19884 raft_consensus.cc:482] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:01.956997 19884 raft_consensus.cc:504] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8112f3b03f8b4fa783dc249a378c4494" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 44247 } } peers { permanent_uuid: "e136e1d520f34bff94c5170c724628cb" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 38727 } } peers { permanent_uuid: "73a9182a8a374ee99d8d33255d5e46b2" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38393 } }
I20260620 19:04:01.957446 19884 leader_election.cc:288] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247), e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727)
W20260620 19:04:01.958194 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8112f3b03f8b4fa783dc249a378c4494 (127.2.208.130:44247): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
W20260620 19:04:01.958334 19465 leader_election.cc:341] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e136e1d520f34bff94c5170c724628cb (127.2.208.131:38727): Not found: Tablet not found: eded847b9a864ef9b5f6dde5f5357cdf
I20260620 19:04:01.958410 19465 leader_election.cc:302] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73a9182a8a374ee99d8d33255d5e46b2; no voters: 8112f3b03f8b4fa783dc249a378c4494, e136e1d520f34bff94c5170c724628cb
I20260620 19:04:01.958582 19884 raft_consensus.cc:2726] T eded847b9a864ef9b5f6dde5f5357cdf P 73a9182a8a374ee99d8d33255d5e46b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:04:02.307195 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:02.306679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:59051)
0620 19:04:02.306971 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:02.307066 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":128,"threads_started":1}
I20260620 19:04:02.461671 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:02.461315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:35789)
0620 19:04:02.461526 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:02.461577 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260620 19:04:02.638320 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:02.637940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:45153)
0620 19:04:02.638192 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:02.638239 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260620 19:04:03.308307 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:03.307928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:58455)
0620 19:04:03.308161 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:03.308212 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260620 19:04:03.462750 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:03.462365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:45305)
0620 19:04:03.462603 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:03.462656 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260620 19:04:03.639340 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:03.638956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:57189)
0620 19:04:03.639195 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:03.639250 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260620 19:04:04.309505 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:04.309097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:35239)
0620 19:04:04.309355 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:04.309409 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
I20260620 19:04:04.463840 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:04.463476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:57651)
0620 19:04:04.463698 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:04.463753 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 19:04:04.640413 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:04.640063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:36511)
0620 19:04:04.640276 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:04.640329 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260620 19:04:05.310626 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:05.310232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:50021)
0620 19:04:05.310481 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:05.310532 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260620 19:04:05.464864 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:05.464492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:44907)
0620 19:04:05.464716 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:05.464774 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260620 19:04:05.641732 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:05.641283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:36383)
0620 19:04:05.641561 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:05.641622 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260620 19:04:06.311925 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:06.311507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.129:46695)
0620 19:04:06.311742 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:06.311830 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
I20260620 19:04:06.466054 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:06.465674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.130:45009)
0620 19:04:06.465904 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:06.465956 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260620 19:04:06.637763 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19414
I20260620 19:04:06.643180 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:06.642733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:36293 (local address 127.2.208.131:39095)
0620 19:04:06.643012 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:06.643064 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.190:36293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":118,"threads_started":1}
I20260620 19:04:06.643576 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19581
I20260620 19:04:06.648967 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19715
2026-06-20T19:04:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59422 ms)
[----------] 30 tests from SecurityITest (238017 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:06 dist-test-slave-tsm4 krb5kdc[19902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:06 dist-test-slave-tsm4 krb5kdc[19902](info): set up 2 sockets
Jun 20 19:04:06 dist-test-slave-tsm4 krb5kdc[19902](info): commencing operation
krb5kdc: starting...
W20260620 19:04:07.675102 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:07 dist-test-slave-tsm4 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:07.792445 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44575
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:07.897389 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:07.897676 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:07.897744 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:07.901659 19915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:07.901742 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:07.901777 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:07.901829 19915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:07.901863 19915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:07.907030 19915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:07.908063 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:07.909092 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:04:07.917498 19915 server_base.cc:1034] running on GCE node
W20260620 19:04:07.917485 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:07.917641 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:07.917485 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:07.918371 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:07.919426 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:07.920603 19915 hybrid_clock.cc:648] HybridClock initialized: now 1781982247920581 us; error 47 us; skew 500 ppm
Jun 20 19:04:07 dist-test-slave-tsm4 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982247, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:07.923203 19915 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:07.924021 19915 webserver.cc:460] Webserver started at http://127.2.208.190:41587/ using document root <none> and password file <none>
I20260620 19:04:07.924480 19915 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:07.924536 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:07.924682 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:07.926560 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "c050a12d9ba346aeae452f6abfbf8ddc"
format_stamp: "Formatted at 2026-06-20 19:04:07 on dist-test-slave-tsm4"
server_key: "37440fdd593285c2feba9e03674e776a"
server_key_iv: "6f0e367c3a7497bd03ef84bdf0b272cb"
server_key_version: "encryptionkey@0"
I20260620 19:04:07.927134 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "c050a12d9ba346aeae452f6abfbf8ddc"
format_stamp: "Formatted at 2026-06-20 19:04:07 on dist-test-slave-tsm4"
server_key: "37440fdd593285c2feba9e03674e776a"
server_key_iv: "6f0e367c3a7497bd03ef84bdf0b272cb"
server_key_version: "encryptionkey@0"
I20260620 19:04:07.930807 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 19:04:07.933128 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:07.934285 19915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:07.934381 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "c050a12d9ba346aeae452f6abfbf8ddc"
format_stamp: "Formatted at 2026-06-20 19:04:07 on dist-test-slave-tsm4"
server_key: "37440fdd593285c2feba9e03674e776a"
server_key_iv: "6f0e367c3a7497bd03ef84bdf0b272cb"
server_key_version: "encryptionkey@0"
I20260620 19:04:07.934463 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:07.961508 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:07.964733 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:07.964890 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:07.971874 19915 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44575
I20260620 19:04:07.971962 19982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44575 every 8 connection(s)
I20260620 19:04:07.972851 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:07.975607 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:07.977905 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 19915
I20260620 19:04:07.978003 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:07.978305 2882 external_mini_cluster.cc:1428] Setting key 1d6e25f77318afe8d490b4294d645d40
I20260620 19:04:07.981318 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: Bootstrap starting.
I20260620 19:04:07.983459 19983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:07.984154 19983 log.cc:824] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:07.986066 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: No bootstrap required, opened a new log
Jun 20 19:04:07 dist-test-slave-tsm4 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:07.989508 19983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } }
I20260620 19:04:07.989771 19983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:07.989858 19983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c050a12d9ba346aeae452f6abfbf8ddc, State: Initialized, Role: FOLLOWER
I20260620 19:04:07.990379 19983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } }
I20260620 19:04:07.990531 19983 raft_consensus.cc:388] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:07.990623 19983 raft_consensus.cc:482] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:07.990721 19983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:07.991869 19983 raft_consensus.cc:504] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } }
I20260620 19:04:07.992209 19983 leader_election.cc:302] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c050a12d9ba346aeae452f6abfbf8ddc; no voters:
I20260620 19:04:07.992525 19983 leader_election.cc:288] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:07.992625 19988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:07.992727 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:07.979665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.2.208.190:44575)
0620 19:04:07.980093 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:07.980107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:07.980141 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:07.980737 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:07.980752 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:07.981074 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:07.981407 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:07.982155 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:07.983089 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:07.984014 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:07.984521 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:07.984686 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:07.987254 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:07.987259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:07.987271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:07.987295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:07.989527 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:07.989998 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:07.990004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:07.990010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:07.990120 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:07.990428 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:07.990432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:07.990433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:07.990962 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:07.991159 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:07.991485 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 19:04:07.991779 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260620 19:04:07.992841 19988 raft_consensus.cc:686] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [term 1 LEADER]: Becoming Leader. State: Replica: c050a12d9ba346aeae452f6abfbf8ddc, State: Running, Role: LEADER
I20260620 19:04:07.993312 19988 consensus_queue.cc:237] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } }
I20260620 19:04:07.993809 19983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:07.995002 19989 sys_catalog.cc:455] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } } }
I20260620 19:04:07.995788 19989 sys_catalog.cc:458] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:07.995653 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c050a12d9ba346aeae452f6abfbf8ddc. Latest consensus state: current_term: 1 leader_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c050a12d9ba346aeae452f6abfbf8ddc" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44575 } } }
I20260620 19:04:07.996083 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:07.996363 20002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:07.998922 20002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:08.003720 20002 catalog_manager.cc:1347] Generated new cluster ID: 34befc5911fc4d43abbf326830334b31
I20260620 19:04:08.003778 20002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:08.026268 20002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:08.027554 20002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:08.035408 20002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: Generated new TSK 0
I20260620 19:04:08.036180 20002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:04:08.064177 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:08.173637 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:08.173884 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:08.173925 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:08.177822 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:08.177874 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:08.177929 20009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:08.182962 20009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:08.183995 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:08.184947 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:08.194183 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:08.194391 20009 server_base.cc:1034] running on GCE node
W20260620 19:04:08.194170 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:08.194160 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:08.195039 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:08.195730 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:08.196890 20009 hybrid_clock.cc:648] HybridClock initialized: now 1781982248196858 us; error 46 us; skew 500 ppm
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:08.199565 20009 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:04:08.200361 20009 webserver.cc:460] Webserver started at http://127.2.208.129:39417/ using document root <none> and password file <none>
I20260620 19:04:08.200922 20009 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:08.201014 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:08.201236 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:08.203269 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "6e6c03bd08e4499c99ced96e79c6bf81"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "c17cf4eefce4d6b82d5c1a1f9d4b963c"
server_key_iv: "baa0402a72307e9200d8132546e56aa2"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.203892 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "6e6c03bd08e4499c99ced96e79c6bf81"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "c17cf4eefce4d6b82d5c1a1f9d4b963c"
server_key_iv: "baa0402a72307e9200d8132546e56aa2"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.207463 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:08.209782 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.210959 20009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:08.211097 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "6e6c03bd08e4499c99ced96e79c6bf81"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "c17cf4eefce4d6b82d5c1a1f9d4b963c"
server_key_iv: "baa0402a72307e9200d8132546e56aa2"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.211205 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:08.222630 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:08.225355 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:08.225549 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:08.226100 20009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:08.226907 20009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:08.226977 20009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.227048 20009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:08.227100 20009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.235911 20009 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:44859
I20260620 19:04:08.235941 20137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:44859 every 8 connection(s)
I20260620 19:04:08.236828 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:08.239950 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20009
I20260620 19:04:08.240067 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:08.240362 2882 external_mini_cluster.cc:1428] Setting key eb56dec4d6cefc9207763035b761bc16
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:04:08.250445 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.238656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:37415 (local address 127.2.208.190:44575)
0620 19:04:08.238839 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.238844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.239693 (+ 849us) server_negotiation.cc:407] Connection header received
0620 19:04:08.240700 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.240703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.240761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.240858 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:08.242008 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.242558 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.244164 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.244395 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.244489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.246889 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:08.246893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:08.246895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:08.246917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:08.248410 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.248933 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.248938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.248940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.249024 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.249351 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.249355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.249357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.249532 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:08.249639 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.250131 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.250273 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:04:08.251219 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.239060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44575 (local address 127.2.208.129:37415)
0620 19:04:08.239540 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.239581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.240491 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.241060 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.241067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.241402 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:08.241856 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.241869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.242709 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.242713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.244026 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.244036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.244542 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.244545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.244716 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.245323 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:08.245338 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:08.246716 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:08.248524 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.248529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.248541 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.248830 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.249115 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.249119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.249121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.249257 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.249637 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:08.249644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:08.249910 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.250093 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.250391 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:04:08.253049 20138 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44575
I20260620 19:04:08.253289 20138 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:08.253924 20138 heartbeater.cc:511] Master 127.2.208.190:44575 requested a full tablet report, sending...
I20260620 19:04:08.255580 19947 ts_manager.cc:194] Registered new tserver with Master: 6e6c03bd08e4499c99ced96e79c6bf81 (127.2.208.129:44859)
I20260620 19:04:08.256690 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:37415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:04:08.267216 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:08.376338 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:08.376586 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:08.376631 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:08.380625 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:08.380679 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:08.380736 20143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:08.385948 20143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:08.386982 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:08.388025 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:08.397743 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:08.397749 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:08.397754 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:08.397791 20143 server_base.cc:1034] running on GCE node
I20260620 19:04:08.398438 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:08.399004 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:08.400158 20143 hybrid_clock.cc:648] HybridClock initialized: now 1781982248400141 us; error 28 us; skew 500 ppm
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:08.402815 20143 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:04:08.403703 20143 webserver.cc:460] Webserver started at http://127.2.208.130:46753/ using document root <none> and password file <none>
I20260620 19:04:08.404253 20143 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:08.404340 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:08.404561 20143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:08.406555 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "a2de64997fb64df1a80f5549d72829f1"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "051c84815bc8b85d2c1294ec9826adfc"
server_key_iv: "d75add3cc436bd53029a1c498cc0c92a"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.407112 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "a2de64997fb64df1a80f5549d72829f1"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "051c84815bc8b85d2c1294ec9826adfc"
server_key_iv: "d75add3cc436bd53029a1c498cc0c92a"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.410698 20143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:04:08.413065 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.414247 20143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:08.414367 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "a2de64997fb64df1a80f5549d72829f1"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "051c84815bc8b85d2c1294ec9826adfc"
server_key_iv: "d75add3cc436bd53029a1c498cc0c92a"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.414469 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:08.429585 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:08.432564 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:08.432801 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:08.433360 20143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:08.434128 20143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:08.434195 20143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.434263 20143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:08.434314 20143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.442958 20143 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:33971
I20260620 19:04:08.442965 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:33971 every 8 connection(s)
I20260620 19:04:08.443854 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:08.453259 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20143
I20260620 19:04:08.453401 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:08.453791 2882 external_mini_cluster.cc:1428] Setting key 2f36aeab71e292770638bec6b20c87d6
I20260620 19:04:08.456936 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.445576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:48165 (local address 127.2.208.190:44575)
0620 19:04:08.445699 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.445702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.446445 (+ 743us) server_negotiation.cc:407] Connection header received
0620 19:04:08.447350 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.447354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.447432 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.447538 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:08.448638 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.449205 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.450322 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.450543 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.450652 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.453080 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:08.453085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:08.453091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:08.453121 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:08.454692 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.455315 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.455318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.455320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.455425 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.455756 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.455760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.455762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.456006 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:08.456113 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.456564 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.456698 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:04:08.457616 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.445884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44575 (local address 127.2.208.130:48165)
0620 19:04:08.446309 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.446348 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.447144 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.447709 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.447716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.448053 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:08.448493 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.448505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.449334 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.449337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.450191 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.450200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.450687 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.450691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.450852 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.451349 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:08.451405 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:08.452898 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:08.454803 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.454809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.454824 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.455200 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.455527 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.455529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.455530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.455662 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.456109 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:08.456114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:08.456339 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.456488 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.456722 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260620 19:04:08.458719 20272 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44575
I20260620 19:04:08.458978 20272 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:08.459504 20272 heartbeater.cc:511] Master 127.2.208.190:44575 requested a full tablet report, sending...
I20260620 19:04:08.460512 19947 ts_manager.cc:194] Registered new tserver with Master: a2de64997fb64df1a80f5549d72829f1 (127.2.208.130:33971)
I20260620 19:04:08.461064 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:48165
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:04:08.480526 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:08.586001 20277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:08.586275 20277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:08.586356 20277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:08.590384 20277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:08.590466 20277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:08.590551 20277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:08.595888 20277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:08.596900 20277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:08.597872 20277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:08.606932 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:08.607081 20277 server_base.cc:1034] running on GCE node
W20260620 19:04:08.606956 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:08.606984 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:08.607633 20277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:08.608153 20277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:08.609308 20277 hybrid_clock.cc:648] HybridClock initialized: now 1781982248609290 us; error 30 us; skew 500 ppm
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:08.611967 20277 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:04:08.612758 20277 webserver.cc:460] Webserver started at http://127.2.208.131:36935/ using document root <none> and password file <none>
I20260620 19:04:08.613227 20277 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:08.613291 20277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:08.613440 20277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:08.615257 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "552fecdcf8fa462293aa71fa1d46bbf8"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "cdc8ea14a21bdeeff8027a47289a4c5c"
server_key_iv: "ff5f78752c67c10c37c37ecd4f1e84f6"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.615820 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "552fecdcf8fa462293aa71fa1d46bbf8"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "cdc8ea14a21bdeeff8027a47289a4c5c"
server_key_iv: "ff5f78752c67c10c37c37ecd4f1e84f6"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.619302 20277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:08.621554 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.622749 20277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:04:08.622902 20277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "552fecdcf8fa462293aa71fa1d46bbf8"
format_stamp: "Formatted at 2026-06-20 19:04:08 on dist-test-slave-tsm4"
server_key: "cdc8ea14a21bdeeff8027a47289a4c5c"
server_key_iv: "ff5f78752c67c10c37c37ecd4f1e84f6"
server_key_version: "encryptionkey@0"
I20260620 19:04:08.623010 20277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:08.645643 20277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:08.648904 20277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:08.649109 20277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:08.649672 20277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:08.650457 20277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:08.650523 20277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.650590 20277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:08.650636 20277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:08.659209 20277 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:40951
I20260620 19:04:08.659235 20405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:40951 every 8 connection(s)
I20260620 19:04:08.660055 20277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:08.666270 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20277
I20260620 19:04:08.666420 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:08.666710 2882 external_mini_cluster.cc:1428] Setting key e7e2c03e8831f4c5d228506d02b06676
Jun 20 19:04:08 dist-test-slave-tsm4 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982248, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:08.673372 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.661874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:37523 (local address 127.2.208.190:44575)
0620 19:04:08.662027 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.662030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.663034 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 19:04:08.664069 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.664072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.664134 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.664252 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:08.665383 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.665902 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.667249 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.667574 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.667679 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.670309 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:08.670312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:08.670314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:08.670338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:08.671748 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.672186 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.672188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.672190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.672264 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.672538 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.672540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.672541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.672692 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:08.672766 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.673163 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.673260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:04:08.674154 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.662210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44575 (local address 127.2.208.131:37523)
0620 19:04:08.662880 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.662919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.663808 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.664416 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.664423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.664774 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:08.665242 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.665255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.666025 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.666029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.667070 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.667081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.667735 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.667748 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.667914 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.668536 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:08.668555 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:08.670175 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:08.671845 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.671849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.671860 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.672103 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.672338 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:08.672340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:08.672341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:08.672465 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:08.672772 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:08.672776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:08.672979 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.673121 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.673343 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":113,"threads_started":1}
I20260620 19:04:08.675271 20406 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44575
I20260620 19:04:08.675522 20406 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:08.675971 20406 heartbeater.cc:511] Master 127.2.208.190:44575 requested a full tablet report, sending...
I20260620 19:04:08.676998 19947 ts_manager.cc:194] Registered new tserver with Master: 552fecdcf8fa462293aa71fa1d46bbf8 (127.2.208.131:40951)
I20260620 19:04:08.677544 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:37523
I20260620 19:04:08.680656 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:08.689038 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.682449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.2.208.190:44575)
0620 19:04:08.682553 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.682557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.682721 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:04:08.682823 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.682825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.682860 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.682925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:08.683657 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.684146 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.684908 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.685099 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.685170 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.686491 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:08.686493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:08.686495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:08.686512 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:08.687733 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.688077 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.688079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.688081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.688147 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:08.688358 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:08.688361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:08.688362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:08.688542 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:08.688638 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.688852 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.688933 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:04:08.691692 19947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:08.694952 19947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:08.709406 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.703008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:33971 (local address 127.0.0.1:33600)
0620 19:04:08.703337 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.703418 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.703536 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.704603 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.704608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.704631 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:08.704759 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.704766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.706448 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.706450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.707673 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.707682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.708809 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.708813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.708958 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.708997 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.709111 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.709241 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":45,"thread_start_us":102,"threads_started":1}
I20260620 19:04:08.709438 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.703703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.2.208.130:33971)
0620 19:04:08.704054 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.704066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.704089 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:04:08.704187 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.704197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.704428 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.704923 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:08.705083 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.706309 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.707793 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.708647 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.708816 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.708967 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.709154 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.709242 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":145,"threads_started":1}
I20260620 19:04:08.710383 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.703327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:40951 (local address 127.0.0.1:52570)
0620 19:04:08.704162 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.704175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.704254 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.705051 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.705054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.705067 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:08.705133 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.705142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.706195 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.706201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.707681 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.707690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.708815 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.708818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.709995 (+ 1177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.710021 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.710085 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.710177 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":1004,"thread_start_us":105,"threads_started":1}
I20260620 19:04:08.710487 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.703812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.2.208.131:40951)
0620 19:04:08.704616 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.704627 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.704649 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:04:08.704724 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.704729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.704887 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.705278 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:08.705392 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.706065 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.707844 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.708647 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.708938 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.709066 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.710262 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.710359 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":118,"threads_started":1}
I20260620 19:04:08.712256 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.704207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.2.208.129:44859)
0620 19:04:08.704504 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.704517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.704540 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:04:08.704612 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.704617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.704808 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.704947 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:08.705214 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.706332 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.708521 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.709315 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.709415 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.709556 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.712003 (+ 2447us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.712101 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260620 19:04:08.712473 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.703045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:44859 (local address 127.0.0.1:50308)
0620 19:04:08.703867 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.703881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.703986 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.704955 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.704961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.704984 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:08.705097 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.705103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.707586 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.707591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.708399 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.708408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.709619 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.709624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.709750 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.709994 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.710078 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.710177 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":192,"thread_start_us":105,"threads_started":1}
I20260620 19:04:08.712265 20206 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 (DEFAULT_TABLE table=test-table [id=956cd1a49e83460fb2e4f449edbc7ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:08.713348 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:08.713730 20340 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 (DEFAULT_TABLE table=test-table [id=956cd1a49e83460fb2e4f449edbc7ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:08.714690 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:08.719745 20423 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Bootstrap starting.
I20260620 19:04:08.719981 20072 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 (DEFAULT_TABLE table=test-table [id=956cd1a49e83460fb2e4f449edbc7ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:08.720985 20072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:08.721144 20424 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Bootstrap starting.
I20260620 19:04:08.721884 20423 tablet_bootstrap.cc:654] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:08.722803 20423 log.cc:824] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:08.723126 20424 tablet_bootstrap.cc:654] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:08.724381 20424 log.cc:824] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:08.724913 20423 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: No bootstrap required, opened a new log
I20260620 19:04:08.725131 20423 ts_tablet_manager.cc:1397] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:04:08.726419 20424 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: No bootstrap required, opened a new log
I20260620 19:04:08.726593 20424 ts_tablet_manager.cc:1397] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:04:08.727713 20427 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: Bootstrap starting.
I20260620 19:04:08.729419 20427 tablet_bootstrap.cc:654] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:08.729370 20423 raft_consensus.cc:348] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.729647 20423 raft_consensus.cc:374] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:08.729733 20423 raft_consensus.cc:729] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2de64997fb64df1a80f5549d72829f1, State: Initialized, Role: FOLLOWER
I20260620 19:04:08.730196 20423 consensus_queue.cc:260] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.730321 20424 raft_consensus.cc:348] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.730602 20424 raft_consensus.cc:374] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:08.730614 20427 log.cc:824] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:08.730679 20424 raft_consensus.cc:729] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552fecdcf8fa462293aa71fa1d46bbf8, State: Initialized, Role: FOLLOWER
I20260620 19:04:08.730906 20272 heartbeater.cc:503] Master 127.2.208.190:44575 was elected leader, sending a full tablet report...
I20260620 19:04:08.731218 20424 consensus_queue.cc:260] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.732239 20424 ts_tablet_manager.cc:1428] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:04:08.732239 20423 ts_tablet_manager.cc:1428] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:04:08.732583 20427 tablet_bootstrap.cc:492] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: No bootstrap required, opened a new log
I20260620 19:04:08.732770 20427 ts_tablet_manager.cc:1397] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:04:08.735518 20406 heartbeater.cc:503] Master 127.2.208.190:44575 was elected leader, sending a full tablet report...
I20260620 19:04:08.736280 20427 raft_consensus.cc:348] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.736563 20427 raft_consensus.cc:374] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:08.736624 20427 raft_consensus.cc:729] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6c03bd08e4499c99ced96e79c6bf81, State: Initialized, Role: FOLLOWER
I20260620 19:04:08.737080 20427 consensus_queue.cc:260] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.737716 20138 heartbeater.cc:503] Master 127.2.208.190:44575 was elected leader, sending a full tablet report...
I20260620 19:04:08.738047 20427 ts_tablet_manager.cc:1428] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:04:08.842963 20428 raft_consensus.cc:482] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:08.843201 20428 raft_consensus.cc:504] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.844405 20428 leader_election.cc:288] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e6c03bd08e4499c99ced96e79c6bf81 (127.2.208.129:44859), 552fecdcf8fa462293aa71fa1d46bbf8 (127.2.208.131:40951)
I20260620 19:04:08.849339 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.844875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:36027 (local address 127.2.208.131:40951)
0620 19:04:08.844989 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.844993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.845007 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:08.845135 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.845138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.845181 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.845282 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:08.845563 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.846341 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.847960 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.848584 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.848720 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.848773 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.849135 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.849202 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:04:08.849339 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.844902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:33315 (local address 127.2.208.129:44859)
0620 19:04:08.845032 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.845037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.845280 (+ 243us) server_negotiation.cc:407] Connection header received
0620 19:04:08.845417 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.845419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.845462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.845564 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:08.845780 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.846457 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.847959 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.848584 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.848720 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.848773 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.849135 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.849202 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:04:08.849364 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.844689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:40951 (local address 127.2.208.130:36027)
0620 19:04:08.844896 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.844912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.845011 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.845291 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.845294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.845306 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:08.845441 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.845446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.846441 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.846443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.847835 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.847841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.848723 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.848728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.848944 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.849026 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.849107 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.849213 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":192}
I20260620 19:04:08.849372 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.844689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:44859 (local address 127.2.208.130:33315)
0620 19:04:08.845184 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.845194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:08.845296 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:08.845585 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:08.845587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:08.845600 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:08.845682 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.845685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.846554 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.846557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.847823 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:08.847835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.848723 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.848728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:08.848929 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.849032 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:04:08.849107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:04:08.849213 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":252,"thread_start_us":105,"threads_started":1}
I20260620 19:04:08.850116 20092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9ed72f004542d4b0ba4c3c0f9752f4" candidate_uuid: "a2de64997fb64df1a80f5549d72829f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" is_pre_election: true
I20260620 19:04:08.850090 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9ed72f004542d4b0ba4c3c0f9752f4" candidate_uuid: "a2de64997fb64df1a80f5549d72829f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" is_pre_election: true
I20260620 19:04:08.850428 20360 raft_consensus.cc:2454] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2de64997fb64df1a80f5549d72829f1 in term 0.
I20260620 19:04:08.850423 20092 raft_consensus.cc:2454] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2de64997fb64df1a80f5549d72829f1 in term 0.
I20260620 19:04:08.850957 20161 leader_election.cc:302] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 552fecdcf8fa462293aa71fa1d46bbf8, 6e6c03bd08e4499c99ced96e79c6bf81, a2de64997fb64df1a80f5549d72829f1; no voters:
I20260620 19:04:08.851408 20428 raft_consensus.cc:2781] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:08.851496 20428 raft_consensus.cc:482] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:08.851557 20428 raft_consensus.cc:3037] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:08.852592 20428 raft_consensus.cc:504] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.852958 20428 leader_election.cc:288] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6e6c03bd08e4499c99ced96e79c6bf81 (127.2.208.129:44859), 552fecdcf8fa462293aa71fa1d46bbf8 (127.2.208.131:40951)
I20260620 19:04:08.853263 20092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9ed72f004542d4b0ba4c3c0f9752f4" candidate_uuid: "a2de64997fb64df1a80f5549d72829f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6c03bd08e4499c99ced96e79c6bf81"
I20260620 19:04:08.853294 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9ed72f004542d4b0ba4c3c0f9752f4" candidate_uuid: "a2de64997fb64df1a80f5549d72829f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552fecdcf8fa462293aa71fa1d46bbf8"
I20260620 19:04:08.853361 20092 raft_consensus.cc:3037] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:08.853430 20360 raft_consensus.cc:3037] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:08.854404 20092 raft_consensus.cc:2454] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2de64997fb64df1a80f5549d72829f1 in term 1.
I20260620 19:04:08.854516 20360 raft_consensus.cc:2454] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2de64997fb64df1a80f5549d72829f1 in term 1.
I20260620 19:04:08.854720 20162 leader_election.cc:302] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e6c03bd08e4499c99ced96e79c6bf81, a2de64997fb64df1a80f5549d72829f1; no voters:
I20260620 19:04:08.854946 20428 raft_consensus.cc:2781] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:08.855172 20428 raft_consensus.cc:686] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 1 LEADER]: Becoming Leader. State: Replica: a2de64997fb64df1a80f5549d72829f1, State: Running, Role: LEADER
I20260620 19:04:08.855526 20428 consensus_queue.cc:237] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } }
I20260620 19:04:08.858596 19947 catalog_manager.cc:5495] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2de64997fb64df1a80f5549d72829f1 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "a2de64997fb64df1a80f5549d72829f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2de64997fb64df1a80f5549d72829f1" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 19:04:08.895771 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.892080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.2.208.130:33971)
0620 19:04:08.892187 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:08.892191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:08.892204 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:04:08.892300 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:08.892302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:08.892362 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:08.892459 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:08.892653 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.893201 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.894152 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:08.894368 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:08.894499 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:08.894792 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:08.895199 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:08.895333 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:08.895598 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 19:04:08.895649 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:04:08.901312 20092 raft_consensus.cc:1261] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 6e6c03bd08e4499c99ced96e79c6bf81 [term 1 FOLLOWER]: Refusing update from remote peer a2de64997fb64df1a80f5549d72829f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:08.901306 20360 raft_consensus.cc:1261] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 1 FOLLOWER]: Refusing update from remote peer a2de64997fb64df1a80f5549d72829f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:08.901998 20433 consensus_queue.cc:1035] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "552fecdcf8fa462293aa71fa1d46bbf8" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:08.902168 20428 consensus_queue.cc:1035] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6c03bd08e4499c99ced96e79c6bf81" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:08.908195 20438 mvcc.cc:205] Tried to move back new op lower bound from 7298999291492868096 to 7298999291314626560. Current Snapshot: MvccSnapshot[applied={T|T < 7298999291492868096}]
I20260620 19:04:08.917353 19946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:08.917492 19946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:08.919884 19946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c050a12d9ba346aeae452f6abfbf8ddc: Sending DeleteTablet for 3 replicas of tablet 5a9ed72f004542d4b0ba4c3c0f9752f4
I20260620 19:04:08.920646 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:08 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:33600
I20260620 19:04:08.920732 20340 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:08 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:52570
I20260620 19:04:08.921142 20072 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:08 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:50308
I20260620 19:04:08.921286 20447 tablet_replica.cc:330] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: stopping tablet replica
I20260620 19:04:08.921360 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20009
I20260620 19:04:08.922675 20447 raft_consensus.cc:2229] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:08.923017 20447 raft_consensus.cc:2258] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:08.923657 20446 tablet_replica.cc:330] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: stopping tablet replica
I20260620 19:04:08.923983 20446 raft_consensus.cc:2229] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:08.924355 20446 raft_consensus.cc:2258] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:08.924569 20447 ts_tablet_manager.cc:1905] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:08.925746 20446 ts_tablet_manager.cc:1905] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:08.928413 20446 ts_tablet_manager.cc:1918] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:08.928571 20446 log.cc:1196] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/5a9ed72f004542d4b0ba4c3c0f9752f4
I20260620 19:04:08.928980 20446 ts_tablet_manager.cc:1939] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P a2de64997fb64df1a80f5549d72829f1: Deleting consensus metadata
I20260620 19:04:08.929253 20447 ts_tablet_manager.cc:1918] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:08.929477 20447 log.cc:1196] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/5a9ed72f004542d4b0ba4c3c0f9752f4
I20260620 19:04:08.929865 20447 ts_tablet_manager.cc:1939] T 5a9ed72f004542d4b0ba4c3c0f9752f4 P 552fecdcf8fa462293aa71fa1d46bbf8: Deleting consensus metadata
I20260620 19:04:08.930202 19931 catalog_manager.cc:4841] TS a2de64997fb64df1a80f5549d72829f1 (127.2.208.130:33971): tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 (table test-table [id=956cd1a49e83460fb2e4f449edbc7ce9]) successfully deleted
W20260620 19:04:08.930686 19934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:44859 (error 108)
I20260620 19:04:08.930930 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20143
I20260620 19:04:08.931048 19933 catalog_manager.cc:4841] TS 552fecdcf8fa462293aa71fa1d46bbf8 (127.2.208.131:40951): tablet 5a9ed72f004542d4b0ba4c3c0f9752f4 (table test-table [id=956cd1a49e83460fb2e4f449edbc7ce9]) successfully deleted
I20260620 19:04:08.936239 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:08.935955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:44859 (local address 127.0.0.1:50312)
0620 19:04:08.936100 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:08.936166 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:44859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 19:04:08.936441 19934 catalog_manager.cc:4584] TS 6e6c03bd08e4499c99ced96e79c6bf81 (127.2.208.129:44859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a9ed72f004542d4b0ba4c3c0f9752f4: Network error: Client connection negotiation failed: client connection to 127.2.208.129:44859: connect: Connection refused (error 111)
I20260620 19:04:08.937541 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20277
I20260620 19:04:08.942847 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 19915
2026-06-20T19:04:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2293 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:08Z Disabled control of system clock
I20260620 19:04:08.972107 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39719
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:09.079115 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:09.079396 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:09.079430 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:09.083309 20453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:09.083379 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:09.083438 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:09.083470 20453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:09.083487 20453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:09.088469 20453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:39719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:09.089454 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:09.090725 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:09.098970 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:09.098963 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.099116 20453 server_base.cc:1034] running on GCE node
W20260620 19:04:09.098973 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.099933 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:09.100972 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:09.102172 20453 hybrid_clock.cc:648] HybridClock initialized: now 1781982249102162 us; error 35 us; skew 500 ppm
I20260620 19:04:09.104017 20453 webserver.cc:460] Webserver started at http://127.2.208.190:33307/ using document root <none> and password file <none>
I20260620 19:04:09.104591 20453 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:09.104681 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:09.104889 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:09.106909 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "de6bb94e07e74710bb5b7df058f6a05d"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "31c82a9b4339fef5ece6c0e99a9ed866"
server_key_iv: "d8dec0b71fcf4802567e39c71ca6078d"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.107502 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "de6bb94e07e74710bb5b7df058f6a05d"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "31c82a9b4339fef5ece6c0e99a9ed866"
server_key_iv: "d8dec0b71fcf4802567e39c71ca6078d"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.110930 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:04:09.113201 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.114373 20453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:09.114503 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "de6bb94e07e74710bb5b7df058f6a05d"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "31c82a9b4339fef5ece6c0e99a9ed866"
server_key_iv: "d8dec0b71fcf4802567e39c71ca6078d"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.114602 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:09.132791 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:09.133432 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:09.133618 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:09.140535 20453 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:39719
I20260620 19:04:09.140547 20519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:39719 every 8 connection(s)
I20260620 19:04:09.141431 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:09.144402 20520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:09.147428 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20453
I20260620 19:04:09.147565 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:09.147851 2882 external_mini_cluster.cc:1428] Setting key 1be200b16913d4dfc6cceac3b0b4f24c
I20260620 19:04:09.150445 20520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: Bootstrap starting.
I20260620 19:04:09.152635 20520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:09.153357 20520 log.cc:824] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:09.155269 20520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: No bootstrap required, opened a new log
I20260620 19:04:09.157002 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.149384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.2.208.190:39719)
0620 19:04:09.150015 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.150030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.150064 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:09.150716 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.150733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.151125 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.151457 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:09.151899 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.152890 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.153841 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.154297 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.154461 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.155329 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:09.155334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:09.155346 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:09.155771 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:09.155834 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:09.155854 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:09.155960 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.156102 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.156344 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260620 19:04:09.159173 20520 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } }
I20260620 19:04:09.159509 20520 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:09.159569 20520 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6bb94e07e74710bb5b7df058f6a05d, State: Initialized, Role: FOLLOWER
I20260620 19:04:09.159955 20520 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } }
I20260620 19:04:09.160096 20520 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:09.160181 20520 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:09.160284 20520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:09.161139 20520 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } }
I20260620 19:04:09.161471 20520 leader_election.cc:302] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6bb94e07e74710bb5b7df058f6a05d; no voters:
I20260620 19:04:09.161767 20520 leader_election.cc:288] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:09.161844 20525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:09.162110 20525 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [term 1 LEADER]: Becoming Leader. State: Replica: de6bb94e07e74710bb5b7df058f6a05d, State: Running, Role: LEADER
I20260620 19:04:09.162396 20525 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } }
I20260620 19:04:09.162955 20520 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:09.164022 20527 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6bb94e07e74710bb5b7df058f6a05d. Latest consensus state: current_term: 1 leader_uuid: "de6bb94e07e74710bb5b7df058f6a05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } } }
I20260620 19:04:09.164149 20527 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:09.164465 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de6bb94e07e74710bb5b7df058f6a05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bb94e07e74710bb5b7df058f6a05d" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 39719 } } }
I20260620 19:04:09.164505 20534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:09.165633 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:09.168074 20534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:09.173533 20534 catalog_manager.cc:1347] Generated new cluster ID: 1e8ef6bf018c429da0e0935f1139699e
I20260620 19:04:09.173593 20534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:09.182796 20534 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:09.183729 20534 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:09.194520 20534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: Generated new TSK 0
I20260620 19:04:09.195194 20534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:09.198051 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--builtin_ntp_servers=127.2.208.148:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:09.310179 20544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:09.310453 20544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:09.310519 20544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:09.315073 20544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:09.315150 20544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:09.315289 20544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:09.323254 20544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:09.324460 20544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:09.325866 20544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:09.334919 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:09.334966 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:09.334935 20552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.335152 20544 server_base.cc:1034] running on GCE node
I20260620 19:04:09.335582 20544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:09.336167 20544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:09.337323 20544 hybrid_clock.cc:648] HybridClock initialized: now 1781982249337309 us; error 25 us; skew 500 ppm
I20260620 19:04:09.339071 20544 webserver.cc:460] Webserver started at http://127.2.208.129:33317/ using document root <none> and password file <none>
I20260620 19:04:09.339670 20544 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:09.339759 20544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:09.339982 20544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:09.341887 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "afb36a08d57c43968bed0e341c2a3250"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "e2f772ad213075c5898f71b70f021364"
server_key_iv: "774c20c9c1b2d64db8753c7f078feee8"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.342455 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "afb36a08d57c43968bed0e341c2a3250"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "e2f772ad213075c5898f71b70f021364"
server_key_iv: "774c20c9c1b2d64db8753c7f078feee8"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.346088 20544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 19:04:09.348562 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.349726 20544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:09.349826 20544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "afb36a08d57c43968bed0e341c2a3250"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "e2f772ad213075c5898f71b70f021364"
server_key_iv: "774c20c9c1b2d64db8753c7f078feee8"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.349952 20544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:09.378209 20544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:09.378948 20544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:09.379145 20544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:09.379837 20544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:09.380767 20544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:09.380827 20544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.380899 20544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:09.380931 20544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.390058 20544 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:38707
I20260620 19:04:09.390080 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:38707 every 8 connection(s)
I20260620 19:04:09.390936 20544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:09.394590 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20544
I20260620 19:04:09.394734 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:09.395022 2882 external_mini_cluster.cc:1428] Setting key c8dd58870b1a5fefa3a55b9d2528394e
I20260620 19:04:09.396907 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--builtin_ntp_servers=127.2.208.148:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:09.401154 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.392804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:38229 (local address 127.2.208.190:39719)
0620 19:04:09.392952 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.392955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.393964 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 19:04:09.395695 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.395698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.395761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.395863 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:09.396834 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.397444 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.399032 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.399256 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.399343 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.400647 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:09.400655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:09.400661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:09.400679 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:09.400703 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:09.400717 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:09.400788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.400988 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.401038 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:04:09.402439 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.393194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39719 (local address 127.2.208.129:38229)
0620 19:04:09.393821 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.393860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.395476 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.396037 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.396044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.396126 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:09.396695 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.396709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.397568 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.397571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.398895 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.398909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.399435 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.399442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.399681 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.400330 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:09.400332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:09.400387 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:09.400473 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:09.400821 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:09.400834 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:09.400842 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.400985 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.401342 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260620 19:04:09.403671 20672 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39719
I20260620 19:04:09.403914 20672 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:09.404410 20672 heartbeater.cc:511] Master 127.2.208.190:39719 requested a full tablet report, sending...
I20260620 19:04:09.405835 20484 ts_manager.cc:194] Registered new tserver with Master: afb36a08d57c43968bed0e341c2a3250 (127.2.208.129:38707)
I20260620 19:04:09.406947 20484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.129:38229
W20260620 19:04:09.505146 20675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:09.505419 20675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:09.505489 20675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:09.509498 20675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:09.509570 20675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:09.509657 20675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:09.514407 20675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:09.515455 20675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:09.516812 20675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:09.526091 20680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:09.526171 20683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.526147 20675 server_base.cc:1034] running on GCE node
W20260620 19:04:09.526098 20681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.526708 20675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:09.527341 20675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:09.528586 20675 hybrid_clock.cc:648] HybridClock initialized: now 1781982249528574 us; error 29 us; skew 500 ppm
I20260620 19:04:09.530396 20675 webserver.cc:460] Webserver started at http://127.2.208.130:34217/ using document root <none> and password file <none>
I20260620 19:04:09.530959 20675 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:09.531044 20675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:09.531306 20675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:09.533458 20675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "00376c547ea647fcabcdcf66f0bd32cb"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "a7120ef08a6cd8913ff9aab52e8d2130"
server_key_iv: "d14cb94eafba6cd2ec02c1a835930810"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.534030 20675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "00376c547ea647fcabcdcf66f0bd32cb"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "a7120ef08a6cd8913ff9aab52e8d2130"
server_key_iv: "d14cb94eafba6cd2ec02c1a835930810"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.537662 20675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 19:04:09.540047 20689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.541301 20675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:09.541424 20675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "00376c547ea647fcabcdcf66f0bd32cb"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "a7120ef08a6cd8913ff9aab52e8d2130"
server_key_iv: "d14cb94eafba6cd2ec02c1a835930810"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.541524 20675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:09.567070 20675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:09.567759 20675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:09.567955 20675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:09.568535 20675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:09.569463 20675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:09.569507 20675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.569568 20675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:09.569602 20675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.577898 20675 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:37093
I20260620 19:04:09.577925 20802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:37093 every 8 connection(s)
I20260620 19:04:09.578733 20675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:09.583374 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20675
I20260620 19:04:09.583530 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:09.583814 2882 external_mini_cluster.cc:1428] Setting key 8d3824daa046f2bb15d3809f04a70b1a
I20260620 19:04:09.585681 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--builtin_ntp_servers=127.2.208.148:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:09.588356 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.580461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59955 (local address 127.2.208.190:39719)
0620 19:04:09.580588 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.580592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.581609 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 19:04:09.583077 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.583081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.583136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.583248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:09.584273 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.584884 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.586241 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.586630 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.586740 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.587818 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:09.587824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:09.587828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:09.587844 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:09.587870 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:09.587882 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:09.587965 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.588118 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.588182 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:04:09.589134 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.580834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39719 (local address 127.2.208.130:59955)
0620 19:04:09.581452 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.581492 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.582853 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.583461 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.583468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.583578 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:09.584130 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.584143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.585023 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.585027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.586114 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.586124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.586769 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.586772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.586924 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.587571 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:09.587572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:09.587610 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:09.587670 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:09.587972 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:09.587978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:09.587982 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.588085 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.588295 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260620 19:04:09.590271 20803 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39719
I20260620 19:04:09.590531 20803 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:09.591157 20803 heartbeater.cc:511] Master 127.2.208.190:39719 requested a full tablet report, sending...
I20260620 19:04:09.592347 20484 ts_manager.cc:194] Registered new tserver with Master: 00376c547ea647fcabcdcf66f0bd32cb (127.2.208.130:37093)
I20260620 19:04:09.593219 20484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.130:59955
W20260620 19:04:09.696933 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:09.697166 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:09.697201 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:09.701027 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:09.701071 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:09.701120 20806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:09.705770 20806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:09.706748 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:09.708051 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:09.717080 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.717264 20806 server_base.cc:1034] running on GCE node
W20260620 19:04:09.717105 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:09.717487 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:09.717919 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:09.718585 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:09.719769 20806 hybrid_clock.cc:648] HybridClock initialized: now 1781982249719753 us; error 26 us; skew 500 ppm
I20260620 19:04:09.721536 20806 webserver.cc:460] Webserver started at http://127.2.208.131:32941/ using document root <none> and password file <none>
I20260620 19:04:09.722103 20806 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:09.722191 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:09.722421 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:09.724453 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "0f155217f3364f149ab984deea8c00e5"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "cb70d72c3d59d8f08c41b7beb9bd2d24"
server_key_iv: "1818e677128141ce04a377e6fd05e373"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.725044 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "0f155217f3364f149ab984deea8c00e5"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "cb70d72c3d59d8f08c41b7beb9bd2d24"
server_key_iv: "1818e677128141ce04a377e6fd05e373"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.728631 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:09.731060 20820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.732270 20806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:09.732409 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "0f155217f3364f149ab984deea8c00e5"
format_stamp: "Formatted at 2026-06-20 19:04:09 on dist-test-slave-tsm4"
server_key: "cb70d72c3d59d8f08c41b7beb9bd2d24"
server_key_iv: "1818e677128141ce04a377e6fd05e373"
server_key_version: "encryptionkey@0"
I20260620 19:04:09.732517 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:09.746805 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:09.747512 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:09.747705 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:09.748246 20806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:09.749140 20806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:09.749209 20806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.749272 20806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:09.749330 20806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:09.758188 20806 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:43067
I20260620 19:04:09.758215 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:43067 every 8 connection(s)
I20260620 19:04:09.758998 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:09.762246 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20806
I20260620 19:04:09.762367 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:09.762602 2882 external_mini_cluster.cc:1428] Setting key e15afd061773f2daa66b9d949397070e
I20260620 19:04:09.768080 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.760758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:48563 (local address 127.2.208.190:39719)
0620 19:04:09.760910 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.760914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.761863 (+ 949us) server_negotiation.cc:407] Connection header received
0620 19:04:09.763462 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.763465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.763518 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.763597 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:09.764534 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.765179 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.766365 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.766576 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.766656 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.767699 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:09.767702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:09.767704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:09.767714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:09.767729 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:09.767736 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:09.767786 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.767916 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.767965 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:04:09.768913 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.761120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:39719 (local address 127.2.208.131:48563)
0620 19:04:09.761714 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.761750 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.763183 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.763768 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.763776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.763855 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:09.764405 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.764418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.765321 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.765327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.766230 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.766239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.766698 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.766700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.766838 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.767475 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:09.767477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:09.767513 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:09.767572 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:09.767804 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:09.767810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:09.767813 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.767903 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.768108 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260620 19:04:09.769891 20934 heartbeater.cc:347] Connected to a master server at 127.2.208.190:39719
I20260620 19:04:09.770117 20934 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:09.770586 20934 heartbeater.cc:511] Master 127.2.208.190:39719 requested a full tablet report, sending...
I20260620 19:04:09.771539 20484 ts_manager.cc:194] Registered new tserver with Master: 0f155217f3364f149ab984deea8c00e5 (127.2.208.131:43067)
I20260620 19:04:09.772058 20484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.131:48563
I20260620 19:04:09.776407 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:09.781688 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.778304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.2.208.190:39719)
0620 19:04:09.778460 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.778465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.778612 (+ 147us) server_negotiation.cc:407] Connection header received
0620 19:04:09.778701 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.778703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.778739 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.778810 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:09.779059 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.779606 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.780397 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.780584 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.780658 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.781321 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:09.781323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:09.781325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:09.781335 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:09.781350 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:09.781357 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:09.781411 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.781480 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.781517 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 19:04:09.784339 20484 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:09.787204 20484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:09.800554 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.795016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38707 (local address 127.0.0.1:46536)
0620 19:04:09.795497 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.795548 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.795672 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.796348 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.796353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.796376 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:09.796508 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.796516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.797966 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.797970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.799012 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.799022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.800045 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.800049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.800149 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.800183 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.800278 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.800390 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":75,"threads_started":1}
I20260620 19:04:09.800560 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.795210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.2.208.129:38707)
0620 19:04:09.795945 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.795959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.795988 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:04:09.796050 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.796054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.796201 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.796360 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:09.796643 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.797840 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.799138 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.799928 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.800047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.800239 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.800303 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.800390 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":105,"threads_started":1}
I20260620 19:04:09.801108 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.795746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37093 (local address 127.0.0.1:41746)
0620 19:04:09.795973 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.795989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.796072 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.797268 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.797271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.797287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:09.797357 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.797363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.798532 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.798536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.799662 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.799673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.800655 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.800660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.800785 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.800833 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.800892 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.800972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":17,"thread_start_us":82,"threads_started":1}
I20260620 19:04:09.801321 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.796481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.2.208.130:37093)
0620 19:04:09.796878 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.796890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.796910 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:04:09.796984 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.796990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.797159 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.797307 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:09.797435 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.798358 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.799784 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.800523 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.800830 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.801008 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.801055 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.801145 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":105,"threads_started":1}
I20260620 19:04:09.801703 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.795121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:43067 (local address 127.0.0.1:35688)
0620 19:04:09.795600 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:09.795612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:09.795672 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:09.797008 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:09.797011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:09.797027 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:09.797116 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.797122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.798356 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.798362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.799591 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:09.799602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.800783 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.800786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:09.801421 (+ 635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.801446 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:04:09.801514 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:04:09.801580 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":74,"threads_started":1}
I20260620 19:04:09.801977 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:09.796058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.2.208.131:43067)
0620 19:04:09.796408 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:09.796422 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:09.796443 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:04:09.796512 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:09.796519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:09.796707 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:09.796867 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:09.797238 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.798247 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.799732 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:09.800515 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:09.801499 (+ 984us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:09.801671 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:09.801720 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:04:09.801819 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":185,"threads_started":1}
I20260620 19:04:09.803843 20606 tablet_service.cc:1467] Processing CreateTablet for tablet 1accbe51ef114b7d96ae248a980edf1f (DEFAULT_TABLE table=test-table [id=192c912d5fc04176a0606447e91e9896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:09.804695 20606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1accbe51ef114b7d96ae248a980edf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:09.804320 20737 tablet_service.cc:1467] Processing CreateTablet for tablet 1accbe51ef114b7d96ae248a980edf1f (DEFAULT_TABLE table=test-table [id=192c912d5fc04176a0606447e91e9896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:09.804320 20868 tablet_service.cc:1467] Processing CreateTablet for tablet 1accbe51ef114b7d96ae248a980edf1f (DEFAULT_TABLE table=test-table [id=192c912d5fc04176a0606447e91e9896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:09.805279 20737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1accbe51ef114b7d96ae248a980edf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:09.805279 20868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1accbe51ef114b7d96ae248a980edf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:09.810513 20951 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Bootstrap starting.
I20260620 19:04:09.812003 20952 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Bootstrap starting.
I20260620 19:04:09.812552 20953 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Bootstrap starting.
I20260620 19:04:09.813771 20951 tablet_bootstrap.cc:654] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:09.813884 20952 tablet_bootstrap.cc:654] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:09.814209 20953 tablet_bootstrap.cc:654] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:09.814656 20951 log.cc:824] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:09.814656 20952 log.cc:824] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:09.814895 20953 log.cc:824] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:09.816648 20952 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: No bootstrap required, opened a new log
I20260620 19:04:09.816648 20951 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: No bootstrap required, opened a new log
I20260620 19:04:09.816663 20953 tablet_bootstrap.cc:492] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: No bootstrap required, opened a new log
I20260620 19:04:09.816833 20952 ts_tablet_manager.cc:1397] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:04:09.816833 20951 ts_tablet_manager.cc:1397] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 19:04:09.816846 20953 ts_tablet_manager.cc:1397] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 19:04:09.820597 20953 raft_consensus.cc:348] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.820822 20953 raft_consensus.cc:374] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:09.820871 20953 raft_consensus.cc:729] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00376c547ea647fcabcdcf66f0bd32cb, State: Initialized, Role: FOLLOWER
I20260620 19:04:09.820957 20952 raft_consensus.cc:348] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.821199 20952 raft_consensus.cc:374] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:09.821259 20952 raft_consensus.cc:729] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f155217f3364f149ab984deea8c00e5, State: Initialized, Role: FOLLOWER
I20260620 19:04:09.821166 20951 raft_consensus.cc:348] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.821396 20951 raft_consensus.cc:374] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:09.821326 20953 consensus_queue.cc:260] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.821452 20951 raft_consensus.cc:729] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb36a08d57c43968bed0e341c2a3250, State: Initialized, Role: FOLLOWER
I20260620 19:04:09.821660 20952 consensus_queue.cc:260] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.821808 20951 consensus_queue.cc:260] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:09.822322 20934 heartbeater.cc:503] Master 127.2.208.190:39719 was elected leader, sending a full tablet report...
I20260620 19:04:09.822343 20953 ts_tablet_manager.cc:1428] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:04:09.822592 20952 ts_tablet_manager.cc:1428] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 19:04:09.822678 20803 heartbeater.cc:503] Master 127.2.208.190:39719 was elected leader, sending a full tablet report...
I20260620 19:04:09.823186 20951 ts_tablet_manager.cc:1428] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 19:04:09.823489 20672 heartbeater.cc:503] Master 127.2.208.190:39719 was elected leader, sending a full tablet report...
W20260620 19:04:09.829525 20804 tablet.cc:2369] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:04:09.892125 20673 tablet.cc:2369] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:04:10.009951 20935 tablet.cc:2369] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:10.107690 20958 raft_consensus.cc:482] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:10.107924 20958 raft_consensus.cc:504] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:10.109117 20958 leader_election.cc:288] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afb36a08d57c43968bed0e341c2a3250 (127.2.208.129:38707), 00376c547ea647fcabcdcf66f0bd32cb (127.2.208.130:37093)
I20260620 19:04:10.112692 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.109528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:38769 (local address 127.2.208.130:37093)
0620 19:04:10.109659 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:10.109663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:10.109678 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:04:10.109771 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:10.109774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:10.109826 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:10.109905 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:10.110208 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.110712 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.111791 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.112170 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.112306 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:10.112360 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:10.112516 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:04:10.112584 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:04:10.112692 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.109393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:37093 (local address 127.2.208.131:38769)
0620 19:04:10.109544 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:10.109562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:10.109671 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:10.109927 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:10.109931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:10.109948 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:10.110086 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:10.110091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.110827 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.110832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:10.111673 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:10.111681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.112287 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.112291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:10.112387 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:10.112415 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:04:10.112500 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:04:10.112566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"spinlock_wait_cycles":31488}
I20260620 19:04:10.112926 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.109524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:36615 (local address 127.2.208.129:38707)
0620 19:04:10.109669 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:10.109675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:10.109951 (+ 276us) server_negotiation.cc:407] Connection header received
0620 19:04:10.110124 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:10.110133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:10.110181 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:10.110265 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:10.110536 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.111023 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.112031 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.112401 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.112525 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:10.112568 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:10.112741 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:04:10.112804 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:04:10.112942 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.109393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38707 (local address 127.2.208.131:36615)
0620 19:04:10.109853 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:10.109864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:10.110015 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:10.110275 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:10.110278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:10.110293 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:10.110425 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:10.110430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.111137 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.111143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:10.111910 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:10.111917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.112519 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.112523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:10.112616 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:10.112642 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:04:10.112726 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:04:10.112787 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":134,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260620 19:04:10.113479 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1accbe51ef114b7d96ae248a980edf1f" candidate_uuid: "0f155217f3364f149ab984deea8c00e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00376c547ea647fcabcdcf66f0bd32cb" is_pre_election: true
I20260620 19:04:10.113529 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1accbe51ef114b7d96ae248a980edf1f" candidate_uuid: "0f155217f3364f149ab984deea8c00e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb36a08d57c43968bed0e341c2a3250" is_pre_election: true
I20260620 19:04:10.113775 20757 raft_consensus.cc:2454] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f155217f3364f149ab984deea8c00e5 in term 0.
I20260620 19:04:10.113838 20626 raft_consensus.cc:2454] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f155217f3364f149ab984deea8c00e5 in term 0.
I20260620 19:04:10.114289 20823 leader_election.cc:302] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00376c547ea647fcabcdcf66f0bd32cb, 0f155217f3364f149ab984deea8c00e5, afb36a08d57c43968bed0e341c2a3250; no voters:
I20260620 19:04:10.114517 20958 raft_consensus.cc:2781] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:10.114611 20958 raft_consensus.cc:482] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:10.114667 20958 raft_consensus.cc:3037] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:10.115738 20958 raft_consensus.cc:504] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:10.116101 20958 leader_election.cc:288] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [CANDIDATE]: Term 1 election: Requested vote from peers afb36a08d57c43968bed0e341c2a3250 (127.2.208.129:38707), 00376c547ea647fcabcdcf66f0bd32cb (127.2.208.130:37093)
I20260620 19:04:10.116389 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1accbe51ef114b7d96ae248a980edf1f" candidate_uuid: "0f155217f3364f149ab984deea8c00e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb36a08d57c43968bed0e341c2a3250"
I20260620 19:04:10.116528 20626 raft_consensus.cc:3037] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:10.116561 20757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1accbe51ef114b7d96ae248a980edf1f" candidate_uuid: "0f155217f3364f149ab984deea8c00e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00376c547ea647fcabcdcf66f0bd32cb"
I20260620 19:04:10.116683 20757 raft_consensus.cc:3037] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:10.117651 20757 raft_consensus.cc:2454] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f155217f3364f149ab984deea8c00e5 in term 1.
I20260620 19:04:10.117693 20626 raft_consensus.cc:2454] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f155217f3364f149ab984deea8c00e5 in term 1.
I20260620 19:04:10.117949 20823 leader_election.cc:302] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00376c547ea647fcabcdcf66f0bd32cb, 0f155217f3364f149ab984deea8c00e5; no voters:
I20260620 19:04:10.118098 20958 raft_consensus.cc:2781] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:10.118309 20958 raft_consensus.cc:686] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 1 LEADER]: Becoming Leader. State: Replica: 0f155217f3364f149ab984deea8c00e5, State: Running, Role: LEADER
I20260620 19:04:10.118634 20958 consensus_queue.cc:237] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } }
I20260620 19:04:10.121892 20483 catalog_manager.cc:5495] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f155217f3364f149ab984deea8c00e5 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "0f155217f3364f149ab984deea8c00e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f155217f3364f149ab984deea8c00e5" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 43067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 } health_report { overall_health: UNKNOWN } } }
I20260620 19:04:10.163797 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.160718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.2.208.131:43067)
0620 19:04:10.160843 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:10.160846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:10.160860 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:10.160925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:10.160928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:10.160976 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:10.161057 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:10.161290 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.161766 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.162518 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:10.162708 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:10.162776 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:10.163009 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:10.163475 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:10.163559 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:10.163664 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 19:04:10.163705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:04:10.169462 20757 raft_consensus.cc:1261] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 1 FOLLOWER]: Refusing update from remote peer 0f155217f3364f149ab984deea8c00e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:10.169476 20626 raft_consensus.cc:1261] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 1 FOLLOWER]: Refusing update from remote peer 0f155217f3364f149ab984deea8c00e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:10.170177 20958 consensus_queue.cc:1035] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00376c547ea647fcabcdcf66f0bd32cb" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:10.170394 20961 consensus_queue.cc:1035] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb36a08d57c43968bed0e341c2a3250" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 38707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:10.178020 20965 mvcc.cc:205] Tried to move back new op lower bound from 7298999296687333376 to 7298999296488529920. Current Snapshot: MvccSnapshot[applied={T|T < 7298999296687333376}]
I20260620 19:04:10.178994 20966 mvcc.cc:205] Tried to move back new op lower bound from 7298999296687333376 to 7298999296488529920. Current Snapshot: MvccSnapshot[applied={T|T < 7298999296687333376}]
I20260620 19:04:10.186592 20483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:10.186757 20483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:10.189359 20483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de6bb94e07e74710bb5b7df058f6a05d: Sending DeleteTablet for 3 replicas of tablet 1accbe51ef114b7d96ae248a980edf1f
I20260620 19:04:10.190104 20606 tablet_service.cc:1514] Processing DeleteTablet for tablet 1accbe51ef114b7d96ae248a980edf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:10 UTC) from {username='slave'} at 127.0.0.1:46536
I20260620 19:04:10.190136 20737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1accbe51ef114b7d96ae248a980edf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:10 UTC) from {username='slave'} at 127.0.0.1:41746
I20260620 19:04:10.190580 20975 tablet_replica.cc:330] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250: stopping tablet replica
I20260620 19:04:10.190582 20868 tablet_service.cc:1514] Processing DeleteTablet for tablet 1accbe51ef114b7d96ae248a980edf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:10 UTC) from {username='slave'} at 127.0.0.1:35688
I20260620 19:04:10.190861 20974 tablet_replica.cc:330] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: stopping tablet replica
I20260620 19:04:10.190897 20975 raft_consensus.cc:2229] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:10.191212 20974 raft_consensus.cc:2229] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:10.191216 20975 raft_consensus.cc:2258] T 1accbe51ef114b7d96ae248a980edf1f P afb36a08d57c43968bed0e341c2a3250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:10.191401 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20544
I20260620 19:04:10.191624 20974 raft_consensus.cc:2258] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:10.194659 20974 ts_tablet_manager.cc:1905] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:10.193713 20976 tablet_replica.cc:330] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: stopping tablet replica
I20260620 19:04:10.195739 20976 raft_consensus.cc:2229] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:10.196293 20976 raft_consensus.cc:2258] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:10.197569 20974 ts_tablet_manager.cc:1918] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:10.197665 20974 log.cc:1196] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/1accbe51ef114b7d96ae248a980edf1f
I20260620 19:04:10.197783 20976 ts_tablet_manager.cc:1905] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:10.197975 20974 ts_tablet_manager.cc:1939] T 1accbe51ef114b7d96ae248a980edf1f P 00376c547ea647fcabcdcf66f0bd32cb: Deleting consensus metadata
I20260620 19:04:10.199127 20470 catalog_manager.cc:4841] TS 00376c547ea647fcabcdcf66f0bd32cb (127.2.208.130:37093): tablet 1accbe51ef114b7d96ae248a980edf1f (table test-table [id=192c912d5fc04176a0606447e91e9896]) successfully deleted
W20260620 19:04:10.199543 20469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:38707 (error 108)
I20260620 19:04:10.199782 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20675
I20260620 19:04:10.201380 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:10.201019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:38707 (local address 127.0.0.1:46550)
0620 19:04:10.201183 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:10.201268 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:04:10.203258 20976 ts_tablet_manager.cc:1918] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:10.203385 20976 log.cc:1196] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/1accbe51ef114b7d96ae248a980edf1f
W20260620 19:04:10.203578 20469 catalog_manager.cc:4584] TS afb36a08d57c43968bed0e341c2a3250 (127.2.208.129:38707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1accbe51ef114b7d96ae248a980edf1f: Network error: Client connection negotiation failed: client connection to 127.2.208.129:38707: connect: Connection refused (error 111)
I20260620 19:04:10.203703 20976 ts_tablet_manager.cc:1939] T 1accbe51ef114b7d96ae248a980edf1f P 0f155217f3364f149ab984deea8c00e5: Deleting consensus metadata
I20260620 19:04:10.204869 20470 catalog_manager.cc:4841] TS 0f155217f3364f149ab984deea8c00e5 (127.2.208.131:43067): tablet 1accbe51ef114b7d96ae248a980edf1f (table test-table [id=192c912d5fc04176a0606447e91e9896]) successfully deleted
I20260620 19:04:10.206806 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20806
I20260620 19:04:10.211748 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20453
2026-06-20T19:04:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1268 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:10 dist-test-slave-tsm4 krb5kdc[20980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:10 dist-test-slave-tsm4 krb5kdc[20980](info): set up 2 sockets
Jun 20 19:04:10 dist-test-slave-tsm4 krb5kdc[20980](info): commencing operation
krb5kdc: starting...
W20260620 19:04:11.235205 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:11.362675 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44939
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:11.481901 20993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:11.482179 20993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:11.482263 20993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:11.486207 20993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:11.486281 20993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:11.486320 20993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:11.486393 20993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:11.486429 20993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:11.491605 20993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:11.492528 20993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:11.493482 20993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:11.501461 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:11.501516 20998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:11.501509 20993 server_base.cc:1034] running on GCE node
W20260620 19:04:11.501473 21001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:11.502194 20993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:11.503252 20993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:11.504453 20993 hybrid_clock.cc:648] HybridClock initialized: now 1781982251504427 us; error 38 us; skew 500 ppm
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:11.507118 20993 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:11.508003 20993 webserver.cc:460] Webserver started at http://127.2.208.190:37593/ using document root <none> and password file <none>
I20260620 19:04:11.508535 20993 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:11.508620 20993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:11.508860 20993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:11.510751 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "659614557e04f11baa05a8e4997fe0b1"
server_key_iv: "7931e70e6b2647fccfbc3034ad82d033"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.511302 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "659614557e04f11baa05a8e4997fe0b1"
server_key_iv: "7931e70e6b2647fccfbc3034ad82d033"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.514741 20993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:04:11.517158 21008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:11.518307 20993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:11.518440 20993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "659614557e04f11baa05a8e4997fe0b1"
server_key_iv: "7931e70e6b2647fccfbc3034ad82d033"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.518543 20993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:11.535502 20993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:11.538184 20993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:11.538367 20993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:11.545007 20993 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:44939
I20260620 19:04:11.545027 21060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:44939 every 8 connection(s)
I20260620 19:04:11.545864 20993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:11.548089 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 20993
I20260620 19:04:11.548228 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:11.548518 2882 external_mini_cluster.cc:1428] Setting key 4fbc3e7f542edb31802f82ceb355ca9b
I20260620 19:04:11.548584 21061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:11.555236 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: Bootstrap starting.
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:11.557660 21061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:11.558522 21061 log.cc:824] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:11.560662 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: No bootstrap required, opened a new log
I20260620 19:04:11.562594 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:11.549880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.2.208.190:44939)
0620 19:04:11.550283 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:11.550300 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:11.550340 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:04:11.551230 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:11.551251 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:11.551635 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:11.551968 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:11.552557 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.553914 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.554677 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.555329 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.555493 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:11.558062 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:11.558068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:11.558083 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:11.558111 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:11.560082 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:11.560446 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:11.560451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:11.560455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:11.560573 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:11.560879 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:11.560882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:11.560884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:11.561230 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:11.561420 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:11.561698 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:04:11.561958 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260620 19:04:11.564085 21061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } }
I20260620 19:04:11.564298 21061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:11.564395 21061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f496b155f2e4b16acaeb01eb9ee3bdd, State: Initialized, Role: FOLLOWER
I20260620 19:04:11.564805 21061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } }
I20260620 19:04:11.564942 21061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:11.565004 21061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:11.565104 21061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:11.565876 21061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } }
I20260620 19:04:11.566205 21061 leader_election.cc:302] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f496b155f2e4b16acaeb01eb9ee3bdd; no voters:
I20260620 19:04:11.566469 21061 leader_election.cc:288] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:11.566619 21066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:11.566859 21066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [term 1 LEADER]: Becoming Leader. State: Replica: 3f496b155f2e4b16acaeb01eb9ee3bdd, State: Running, Role: LEADER
I20260620 19:04:11.567240 21066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } }
I20260620 19:04:11.567780 21061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:11.568325 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f496b155f2e4b16acaeb01eb9ee3bdd. Latest consensus state: current_term: 1 leader_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } } }
I20260620 19:04:11.568526 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:11.568732 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f496b155f2e4b16acaeb01eb9ee3bdd" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 44939 } } }
I20260620 19:04:11.568898 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:11.569078 21075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:11.572084 21075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:11.577184 21075 catalog_manager.cc:1347] Generated new cluster ID: df21c27c7b654bde8dad70d35fd25700
I20260620 19:04:11.577243 21075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:11.598647 21075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:11.599882 21075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:11.614545 21075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: Generated new TSK 0
I20260620 19:04:11.615321 21075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:04:11.648618 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:11.756779 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:11.757022 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:11.757069 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:11.760941 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:11.760983 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:11.761037 21087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:11.765985 21087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:11.767009 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:11.768006 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:11.777395 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:11.777437 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:11.777465 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:11.777527 21087 server_base.cc:1034] running on GCE node
I20260620 19:04:11.777963 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:11.778502 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:11.779659 21087 hybrid_clock.cc:648] HybridClock initialized: now 1781982251779643 us; error 26 us; skew 500 ppm
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:11.785506 21087 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:04:11.786327 21087 webserver.cc:460] Webserver started at http://127.2.208.129:34601/ using document root <none> and password file <none>
I20260620 19:04:11.786876 21087 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:11.786962 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:11.787204 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:11.789211 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "3bfb900bed864a629d1c3ce183427533"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "554da328fe9affdf3811b87fe836b757"
server_key_iv: "79d18c00126a3f072df95a551b209f3b"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.789826 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "3bfb900bed864a629d1c3ce183427533"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "554da328fe9affdf3811b87fe836b757"
server_key_iv: "79d18c00126a3f072df95a551b209f3b"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.793491 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:11.795734 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:11.796885 21087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:11.797019 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "3bfb900bed864a629d1c3ce183427533"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "554da328fe9affdf3811b87fe836b757"
server_key_iv: "79d18c00126a3f072df95a551b209f3b"
server_key_version: "encryptionkey@0"
I20260620 19:04:11.797139 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:11.828195 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:11.831249 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:11.831475 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:11.832042 21087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:11.832808 21087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:11.832875 21087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:11.832935 21087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:11.832983 21087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:11.841516 21087 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:33027
I20260620 19:04:11.841552 21215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:33027 every 8 connection(s)
I20260620 19:04:11.842319 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:11.844425 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21087
I20260620 19:04:11.844566 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:11.844859 2882 external_mini_cluster.cc:1428] Setting key 7f678902d4b0d5f5123b9255c21c9d7d
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:04:11.856112 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:11.844104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:57045 (local address 127.2.208.190:44939)
0620 19:04:11.844296 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:11.844303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:11.845069 (+ 766us) server_negotiation.cc:407] Connection header received
0620 19:04:11.846510 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:11.846514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:11.846564 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:11.846649 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:11.847888 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.848442 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.849778 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.850217 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.850344 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:11.852850 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:11.852854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:11.852856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:11.852886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:11.854395 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:11.854838 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:11.854841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:11.854842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:11.854917 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:11.855183 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:11.855185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:11.855186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:11.855416 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:11.855498 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:11.855897 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 19:04:11.855998 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:04:11.856985 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:11.844440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44939 (local address 127.2.208.129:57045)
0620 19:04:11.844913 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:11.844963 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:11.846328 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:11.846791 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:11.846798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:11.847157 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:11.847700 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:11.847735 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.848562 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.848565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:11.849650 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:11.849660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:11.850367 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:11.850370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:11.850496 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:11.851074 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:11.851089 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:11.852698 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:11.854501 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:11.854505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:11.854514 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:11.854735 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:11.854990 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:11.854992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:11.854993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:11.855107 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:11.855501 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:11.855505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:11.855701 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 19:04:11.855854 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 19:04:11.856066 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260620 19:04:11.858156 21216 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44939
I20260620 19:04:11.858382 21216 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:11.858839 21216 heartbeater.cc:511] Master 127.2.208.190:44939 requested a full tablet report, sending...
I20260620 19:04:11.860424 21024 ts_manager.cc:194] Registered new tserver with Master: 3bfb900bed864a629d1c3ce183427533 (127.2.208.129:33027)
I20260620 19:04:11.862079 21024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:57045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:04:11.871995 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:11.976178 21221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:11.976397 21221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:11.976441 21221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:11.980254 21221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:11.980300 21221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:11.980353 21221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:11.985146 21221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:11.985978 21221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:11.986881 21221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:11.995880 21226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:11.995885 21229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:11.996006 21221 server_base.cc:1034] running on GCE node
W20260620 19:04:11.995891 21227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:11.996542 21221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:11.997143 21221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:11.998306 21221 hybrid_clock.cc:648] HybridClock initialized: now 1781982251998280 us; error 27 us; skew 500 ppm
Jun 20 19:04:11 dist-test-slave-tsm4 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:12.001008 21221 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:04:12.001784 21221 webserver.cc:460] Webserver started at http://127.2.208.130:33723/ using document root <none> and password file <none>
I20260620 19:04:12.002279 21221 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:12.002353 21221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:12.002504 21221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:12.004380 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "ce240d5cf6f640fb9a7af65c5acfbb63"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "4b14d1e2aaed7689ce87436fd521b0ec"
server_key_iv: "207ec1da22aa72185729c3f1a2fde3a8"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.004868 21221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "ce240d5cf6f640fb9a7af65c5acfbb63"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "4b14d1e2aaed7689ce87436fd521b0ec"
server_key_iv: "207ec1da22aa72185729c3f1a2fde3a8"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.008345 21221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:12.010579 21236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.011797 21221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:12.011937 21221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "ce240d5cf6f640fb9a7af65c5acfbb63"
format_stamp: "Formatted at 2026-06-20 19:04:11 on dist-test-slave-tsm4"
server_key: "4b14d1e2aaed7689ce87436fd521b0ec"
server_key_iv: "207ec1da22aa72185729c3f1a2fde3a8"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.012043 21221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:12.027588 21221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:12.030237 21221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:12.030368 21221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:12.030802 21221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:12.031518 21221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:12.031569 21221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.031605 21221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:12.031625 21221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.040324 21221 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:35369
I20260620 19:04:12.040366 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:35369 every 8 connection(s)
I20260620 19:04:12.041172 21221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:12.047667 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21221
I20260620 19:04:12.047816 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:12.048125 2882 external_mini_cluster.cc:1428] Setting key 613efbc880c75ca3e4ad6945ff0b9ac6
Jun 20 19:04:12 dist-test-slave-tsm4 krb5kdc[20980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:12.054543 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.042916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:41999 (local address 127.2.208.190:44939)
0620 19:04:12.043044 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.043048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.043831 (+ 783us) server_negotiation.cc:407] Connection header received
0620 19:04:12.045029 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.045031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.045076 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.045147 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:12.046194 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.046716 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.047920 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.048256 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.048366 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.050875 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:12.050882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:12.050888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:12.050929 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:12.052427 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.052997 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.053001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.053003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.053094 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.053428 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.053433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.053434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.053619 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:12.053740 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.054240 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.054378 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:04:12.055260 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.043250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44939 (local address 127.2.208.130:41999)
0620 19:04:12.043681 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.043722 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.044814 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.045270 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.045276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.045605 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:12.046071 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.046083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.046819 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.046821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.047749 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.047759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.048395 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.048398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.048530 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.049132 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:12.049148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:12.050714 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:12.052546 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:12.052550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:12.052561 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:12.052852 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:12.053188 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:12.053191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:12.053193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:12.053328 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:12.053752 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:12.053760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:12.053996 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.054197 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.054498 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":143,"threads_started":1}
I20260620 19:04:12.056545 21350 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44939
I20260620 19:04:12.056769 21350 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:12.057317 21350 heartbeater.cc:511] Master 127.2.208.190:44939 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:04:12.058517 21024 ts_manager.cc:194] Registered new tserver with Master: ce240d5cf6f640fb9a7af65c5acfbb63 (127.2.208.130:35369)
I20260620 19:04:12.059127 21024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:41999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:04:12.074656 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:12.180686 21355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:12.180895 21355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:12.180974 21355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:12.184819 21355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:12.184872 21355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:12.184973 21355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:12.189975 21355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:12.190933 21355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:12.191948 21355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:12.201162 21360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:12.201207 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:12.201179 21361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:12.201388 21355 server_base.cc:1034] running on GCE node
I20260620 19:04:12.201797 21355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:12.202443 21355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:12.203591 21355 hybrid_clock.cc:648] HybridClock initialized: now 1781982252203577 us; error 24 us; skew 500 ppm
Jun 20 19:04:12 dist-test-slave-tsm4 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:12.206067 21355 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:04:12.206893 21355 webserver.cc:460] Webserver started at http://127.2.208.131:41237/ using document root <none> and password file <none>
I20260620 19:04:12.207515 21355 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:12.207603 21355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:12.207816 21355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:12.209776 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "15669153e7be4b5896486fcb0fc1a5fd"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "e5eb4b13ca6b162fde566d7abf4b3487"
server_key_iv: "b8e3f3e4bd56932516420ecbaea69e15"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.210325 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "15669153e7be4b5896486fcb0fc1a5fd"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "e5eb4b13ca6b162fde566d7abf4b3487"
server_key_iv: "b8e3f3e4bd56932516420ecbaea69e15"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.213762 21355 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:04:12.216037 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.217164 21355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:12.217293 21355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "15669153e7be4b5896486fcb0fc1a5fd"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "e5eb4b13ca6b162fde566d7abf4b3487"
server_key_iv: "b8e3f3e4bd56932516420ecbaea69e15"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.217397 21355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:12.233268 21355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:12.235931 21355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:12.236123 21355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:12.236687 21355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:12.237498 21355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:12.237569 21355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.237640 21355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:12.237692 21355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.246572 21355 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:34207
I20260620 19:04:12.246595 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:34207 every 8 connection(s)
I20260620 19:04:12.247493 21355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:12.249867 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21355
I20260620 19:04:12.250000 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:12.250301 2882 external_mini_cluster.cc:1428] Setting key cfc16139e0413c05f47c475095611ead
Jun 20 19:04:12 dist-test-slave-tsm4 krb5kdc[20980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:12.260509 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.249253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:53935 (local address 127.2.208.190:44939)
0620 19:04:12.249404 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.249408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.250321 (+ 913us) server_negotiation.cc:407] Connection header received
0620 19:04:12.252045 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.252047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.252090 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.252156 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:12.253351 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.253867 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.254972 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.255177 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.255249 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.257613 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:12.257615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:12.257617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:12.257636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:12.258824 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.259259 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.259261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.259262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.259330 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.259663 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.259666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.259667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.259819 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:12.259891 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.260309 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.260396 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:04:12.261297 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.249639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:44939 (local address 127.2.208.131:53935)
0620 19:04:12.250169 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.250211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.251837 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.252344 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.252354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.252766 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:12.253225 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.253237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.253994 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.253997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.254874 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.254882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.255294 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.255296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.255459 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.256078 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:12.256094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:12.257477 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:12.258921 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:12.258925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:12.258935 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:12.259163 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:12.259469 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:12.259471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:12.259472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:12.259588 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:12.259901 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:12.259905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:12.260122 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.260263 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.260490 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":157,"threads_started":1}
I20260620 19:04:12.262339 21484 heartbeater.cc:347] Connected to a master server at 127.2.208.190:44939
I20260620 19:04:12.262553 21484 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:12.263070 21484 heartbeater.cc:511] Master 127.2.208.190:44939 requested a full tablet report, sending...
I20260620 19:04:12.264089 21024 ts_manager.cc:194] Registered new tserver with Master: 15669153e7be4b5896486fcb0fc1a5fd (127.2.208.131:34207)
I20260620 19:04:12.264628 21024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:53935
I20260620 19:04:12.275622 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:12.284204 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.277416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.2.208.190:44939)
0620 19:04:12.277531 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.277535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.277720 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:04:12.277809 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.277811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.277844 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.277910 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:12.278535 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.278996 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.279800 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.279994 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.280067 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.281470 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:12.281473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:12.281474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:12.281490 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:12.282607 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.283085 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.283087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.283088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.283153 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:12.283465 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:12.283472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:12.283474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:12.283644 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:12.283755 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.284028 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.284098 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:04:12.286947 21024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:12.290123 21024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:12.304167 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.298234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.2.208.129:33027)
0620 19:04:12.299016 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.299033 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.299056 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:04:12.299131 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.299136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.299350 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.299877 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:12.300013 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.301083 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.302678 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.303471 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.303769 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.303894 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.303946 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.304028 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":128,"threads_started":1}
I20260620 19:04:12.304163 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.298088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33027 (local address 127.0.0.1:53780)
0620 19:04:12.298584 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.298628 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.298743 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.299539 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.299546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.299575 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:12.299728 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.299734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.301222 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.301229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.302551 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.302561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.303594 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.303597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.303688 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.303772 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.303858 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.304000 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":100,"thread_start_us":62,"threads_started":1}
I20260620 19:04:12.304163 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.298243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:35369 (local address 127.0.0.1:36490)
0620 19:04:12.298931 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.298944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.299026 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.299768 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.299771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.299789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:12.299880 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.299885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.301614 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.301617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.302705 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.302712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.303553 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.303556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.303666 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.303759 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.303860 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.304000 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":32,"thread_start_us":124,"threads_started":1}
I20260620 19:04:12.304477 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.298744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.2.208.130:35369)
0620 19:04:12.299285 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.299298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.299320 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:04:12.299420 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.299425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.299617 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.300118 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:12.300321 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.301483 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.302822 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.303430 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.303557 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.303708 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.304225 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.304332 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":113,"threads_started":1}
I20260620 19:04:12.305299 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.298747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:34207 (local address 127.0.0.1:53686)
0620 19:04:12.299531 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.299543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.299606 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.300508 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.300510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.300526 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:12.300618 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.300623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.302188 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.302191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.303447 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.303458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.304917 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.304921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.305001 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.305023 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.305085 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.305144 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":35,"thread_start_us":100,"threads_started":1}
I20260620 19:04:12.305935 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.299139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.2.208.131:34207)
0620 19:04:12.299848 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.299859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.299878 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:04:12.299946 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.299950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.300229 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.300368 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:12.300722 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.302047 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.303585 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.304390 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.304484 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.305671 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.305721 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.305806 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":125,"threads_started":1}
I20260620 19:04:12.307401 21150 tablet_service.cc:1467] Processing CreateTablet for tablet 350007d84b1b44538837c10158f6c7e4 (DEFAULT_TABLE table=test-table [id=e03c7ecd4de140f491ea47e58058557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:12.307448 21284 tablet_service.cc:1467] Processing CreateTablet for tablet 350007d84b1b44538837c10158f6c7e4 (DEFAULT_TABLE table=test-table [id=e03c7ecd4de140f491ea47e58058557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:12.308360 21150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350007d84b1b44538837c10158f6c7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:12.308446 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350007d84b1b44538837c10158f6c7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:12.308521 21418 tablet_service.cc:1467] Processing CreateTablet for tablet 350007d84b1b44538837c10158f6c7e4 (DEFAULT_TABLE table=test-table [id=e03c7ecd4de140f491ea47e58058557a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:12.309417 21418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350007d84b1b44538837c10158f6c7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:12.314847 21501 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Bootstrap starting.
I20260620 19:04:12.316193 21502 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Bootstrap starting.
I20260620 19:04:12.316392 21503 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Bootstrap starting.
I20260620 19:04:12.317167 21501 tablet_bootstrap.cc:654] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:12.318192 21501 log.cc:824] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:12.318284 21503 tablet_bootstrap.cc:654] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:12.318706 21502 tablet_bootstrap.cc:654] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:12.319506 21503 log.cc:824] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:12.319506 21502 log.cc:824] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:12.320204 21501 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: No bootstrap required, opened a new log
I20260620 19:04:12.320403 21501 ts_tablet_manager.cc:1397] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:04:12.321318 21502 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: No bootstrap required, opened a new log
I20260620 19:04:12.321401 21503 tablet_bootstrap.cc:492] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: No bootstrap required, opened a new log
I20260620 19:04:12.321496 21502 ts_tablet_manager.cc:1397] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:04:12.321589 21503 ts_tablet_manager.cc:1397] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260620 19:04:12.324155 21501 raft_consensus.cc:348] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.324416 21501 raft_consensus.cc:374] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:12.324471 21501 raft_consensus.cc:729] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfb900bed864a629d1c3ce183427533, State: Initialized, Role: FOLLOWER
I20260620 19:04:12.324414 21502 raft_consensus.cc:348] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.324698 21502 raft_consensus.cc:374] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:12.324759 21502 raft_consensus.cc:729] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce240d5cf6f640fb9a7af65c5acfbb63, State: Initialized, Role: FOLLOWER
I20260620 19:04:12.324941 21501 consensus_queue.cc:260] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.325179 21502 consensus_queue.cc:260] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.325169 21503 raft_consensus.cc:348] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.325413 21503 raft_consensus.cc:374] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:12.325466 21503 raft_consensus.cc:729] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15669153e7be4b5896486fcb0fc1a5fd, State: Initialized, Role: FOLLOWER
I20260620 19:04:12.325836 21216 heartbeater.cc:503] Master 127.2.208.190:44939 was elected leader, sending a full tablet report...
I20260620 19:04:12.325853 21503 consensus_queue.cc:260] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.325959 21501 ts_tablet_manager.cc:1428] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:04:12.326462 21484 heartbeater.cc:503] Master 127.2.208.190:44939 was elected leader, sending a full tablet report...
I20260620 19:04:12.326973 21503 ts_tablet_manager.cc:1428] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:04:12.328832 21502 ts_tablet_manager.cc:1428] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260620 19:04:12.329138 21350 heartbeater.cc:503] Master 127.2.208.190:44939 was elected leader, sending a full tablet report...
W20260620 19:04:12.343544 21217 tablet.cc:2369] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:04:12.498477 21485 tablet.cc:2369] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:04:12.542482 21351 tablet.cc:2369] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:12.634061 21508 raft_consensus.cc:482] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:12.634313 21508 raft_consensus.cc:504] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.635538 21508 leader_election.cc:288] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfb900bed864a629d1c3ce183427533 (127.2.208.129:33027), 15669153e7be4b5896486fcb0fc1a5fd (127.2.208.131:34207)
I20260620 19:04:12.640758 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.635883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:47609 (local address 127.2.208.129:33027)
0620 19:04:12.636049 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.636054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.636182 (+ 128us) server_negotiation.cc:407] Connection header received
0620 19:04:12.636381 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.636388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.636444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.636536 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:12.636826 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.637631 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.639400 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.640010 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.640143 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.640196 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.640550 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.640627 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:04:12.640758 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.636318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:59211 (local address 127.2.208.131:34207)
0620 19:04:12.636442 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.636446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.636547 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:04:12.636624 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.636627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.636670 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.636733 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:12.637000 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.637724 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.639400 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.640035 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.640143 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.640196 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.640545 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.640627 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:04:12.640795 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.635746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33027 (local address 127.2.208.130:47609)
0620 19:04:12.636080 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.636102 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.636240 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.636533 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.636536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.636548 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:12.636693 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.636698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.637733 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.637735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.639228 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.639239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.640155 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.640159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.640329 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.640401 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.640483 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.640608 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":143,"thread_start_us":133,"threads_started":1}
I20260620 19:04:12.640801 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.636191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:34207 (local address 127.2.208.130:59211)
0620 19:04:12.636446 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.636459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:12.636546 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:12.636773 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:12.636778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:12.636800 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:12.636898 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.636903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.637822 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.637825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.639231 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:12.639239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.640155 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.640159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:12.640350 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.640412 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 19:04:12.640483 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:04:12.640610 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":228,"thread_start_us":91,"threads_started":1}
I20260620 19:04:12.641541 21170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350007d84b1b44538837c10158f6c7e4" candidate_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfb900bed864a629d1c3ce183427533" is_pre_election: true
I20260620 19:04:12.641739 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350007d84b1b44538837c10158f6c7e4" candidate_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15669153e7be4b5896486fcb0fc1a5fd" is_pre_election: true
I20260620 19:04:12.641877 21170 raft_consensus.cc:2454] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce240d5cf6f640fb9a7af65c5acfbb63 in term 0.
I20260620 19:04:12.642055 21438 raft_consensus.cc:2454] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce240d5cf6f640fb9a7af65c5acfbb63 in term 0.
I20260620 19:04:12.642436 21239 leader_election.cc:302] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfb900bed864a629d1c3ce183427533, ce240d5cf6f640fb9a7af65c5acfbb63; no voters:
I20260620 19:04:12.642729 21508 raft_consensus.cc:2781] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:12.642807 21508 raft_consensus.cc:482] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:12.642835 21508 raft_consensus.cc:3037] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:12.643911 21508 raft_consensus.cc:504] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.644236 21508 leader_election.cc:288] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [CANDIDATE]: Term 1 election: Requested vote from peers 3bfb900bed864a629d1c3ce183427533 (127.2.208.129:33027), 15669153e7be4b5896486fcb0fc1a5fd (127.2.208.131:34207)
I20260620 19:04:12.644588 21170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350007d84b1b44538837c10158f6c7e4" candidate_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfb900bed864a629d1c3ce183427533"
I20260620 19:04:12.644680 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350007d84b1b44538837c10158f6c7e4" candidate_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15669153e7be4b5896486fcb0fc1a5fd"
I20260620 19:04:12.644722 21170 raft_consensus.cc:3037] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:12.644833 21438 raft_consensus.cc:3037] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:12.645773 21170 raft_consensus.cc:2454] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce240d5cf6f640fb9a7af65c5acfbb63 in term 1.
I20260620 19:04:12.645824 21438 raft_consensus.cc:2454] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce240d5cf6f640fb9a7af65c5acfbb63 in term 1.
I20260620 19:04:12.646077 21239 leader_election.cc:302] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bfb900bed864a629d1c3ce183427533, ce240d5cf6f640fb9a7af65c5acfbb63; no voters:
I20260620 19:04:12.646306 21508 raft_consensus.cc:2781] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:12.646556 21508 raft_consensus.cc:686] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 1 LEADER]: Becoming Leader. State: Replica: ce240d5cf6f640fb9a7af65c5acfbb63, State: Running, Role: LEADER
I20260620 19:04:12.646906 21508 consensus_queue.cc:237] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } }
I20260620 19:04:12.650072 21023 catalog_manager.cc:5495] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce240d5cf6f640fb9a7af65c5acfbb63 (127.2.208.130). New cstate: current_term: 1 leader_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce240d5cf6f640fb9a7af65c5acfbb63" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 } health_report { overall_health: UNKNOWN } } }
I20260620 19:04:12.664042 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.660603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.2.208.130:35369)
0620 19:04:12.660730 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.660737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.660751 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:12.660948 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.660950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.660997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.661099 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:12.661337 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.661837 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.662664 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.662873 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.662978 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.663238 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:12.663655 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:12.663801 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.663887 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.663920 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:04:12.669466 21438 raft_consensus.cc:1261] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 1 FOLLOWER]: Refusing update from remote peer ce240d5cf6f640fb9a7af65c5acfbb63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:12.669466 21170 raft_consensus.cc:1261] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 1 FOLLOWER]: Refusing update from remote peer ce240d5cf6f640fb9a7af65c5acfbb63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:12.670147 21508 consensus_queue.cc:1035] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfb900bed864a629d1c3ce183427533" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:12.670324 21512 consensus_queue.cc:1035] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15669153e7be4b5896486fcb0fc1a5fd" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 34207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:12.686236 21023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:12.686388 21023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:12.688675 21023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f496b155f2e4b16acaeb01eb9ee3bdd: Sending DeleteTablet for 3 replicas of tablet 350007d84b1b44538837c10158f6c7e4
I20260620 19:04:12.689484 21284 tablet_service.cc:1514] Processing DeleteTablet for tablet 350007d84b1b44538837c10158f6c7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:12 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:36490
I20260620 19:04:12.689514 21150 tablet_service.cc:1514] Processing DeleteTablet for tablet 350007d84b1b44538837c10158f6c7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:12 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:53780
I20260620 19:04:12.689657 21418 tablet_service.cc:1514] Processing DeleteTablet for tablet 350007d84b1b44538837c10158f6c7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:12 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:53686
I20260620 19:04:12.689944 21526 tablet_replica.cc:330] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63: stopping tablet replica
I20260620 19:04:12.690145 21527 tablet_replica.cc:330] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: stopping tablet replica
I20260620 19:04:12.690255 21526 raft_consensus.cc:2229] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:12.690532 21525 tablet_replica.cc:330] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533: stopping tablet replica
I20260620 19:04:12.690586 21526 raft_consensus.cc:2258] T 350007d84b1b44538837c10158f6c7e4 P ce240d5cf6f640fb9a7af65c5acfbb63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:12.690784 21525 raft_consensus.cc:2229] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:12.690904 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21087
I20260620 19:04:12.691066 21525 raft_consensus.cc:2258] T 350007d84b1b44538837c10158f6c7e4 P 3bfb900bed864a629d1c3ce183427533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:12.691417 21527 raft_consensus.cc:2229] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:12.691758 21527 raft_consensus.cc:2258] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:12.693361 21527 ts_tablet_manager.cc:1905] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:12.695703 21527 ts_tablet_manager.cc:1918] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:12.695815 21527 log.cc:1196] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/350007d84b1b44538837c10158f6c7e4
I20260620 19:04:12.696134 21527 ts_tablet_manager.cc:1939] T 350007d84b1b44538837c10158f6c7e4 P 15669153e7be4b5896486fcb0fc1a5fd: Deleting consensus metadata
I20260620 19:04:12.697237 21011 catalog_manager.cc:4841] TS 15669153e7be4b5896486fcb0fc1a5fd (127.2.208.131:34207): tablet 350007d84b1b44538837c10158f6c7e4 (table test-table [id=e03c7ecd4de140f491ea47e58058557a]) successfully deleted
W20260620 19:04:12.699422 21011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:33027 (error 108)
I20260620 19:04:12.699774 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21221
I20260620 19:04:12.701275 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.700823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33027 (local address 127.0.0.1:42442)
0620 19:04:12.701071 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.701159 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260620 19:04:12.701427 21011 catalog_manager.cc:4584] TS 3bfb900bed864a629d1c3ce183427533 (127.2.208.129:33027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 350007d84b1b44538837c10158f6c7e4: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33027: connect: Connection refused (error 111)
I20260620 19:04:12.705439 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21355
I20260620 19:04:12.706513 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.705601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:35369 (local address 127.0.0.1:37312)
0620 19:04:12.706357 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:12.706424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.130:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657}
I20260620 19:04:12.710956 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 20993
2026-06-20T19:04:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2499 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:12Z Disabled control of system clock
I20260620 19:04:12.740069 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34203
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:12.846627 21532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:12.846863 21532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:12.846896 21532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:12.850816 21532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:12.850865 21532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:12.850880 21532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:12.850896 21532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:12.850960 21532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:12.855659 21532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:34203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:12.856585 21532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:12.857851 21532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:04:12.865790 21532 server_base.cc:1034] running on GCE node
W20260620 19:04:12.865721 21538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:12.865777 21537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:12.865754 21540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:12.866448 21532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:12.867492 21532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:12.868649 21532 hybrid_clock.cc:648] HybridClock initialized: now 1781982252868631 us; error 31 us; skew 500 ppm
I20260620 19:04:12.870383 21532 webserver.cc:460] Webserver started at http://127.2.208.190:39611/ using document root <none> and password file <none>
I20260620 19:04:12.870905 21532 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:12.870961 21532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:12.871218 21532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:12.873111 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "991cc131b9344dedaab95991e0d4600e"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "9571afd2fe4a9365a5ffedefbe5702b6"
server_key_iv: "71afca24fd5431f05f4768be9c81ed79"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.873634 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "991cc131b9344dedaab95991e0d4600e"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "9571afd2fe4a9365a5ffedefbe5702b6"
server_key_iv: "71afca24fd5431f05f4768be9c81ed79"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.876963 21532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:12.879141 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:12.880268 21532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:12.880385 21532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "991cc131b9344dedaab95991e0d4600e"
format_stamp: "Formatted at 2026-06-20 19:04:12 on dist-test-slave-tsm4"
server_key: "9571afd2fe4a9365a5ffedefbe5702b6"
server_key_iv: "71afca24fd5431f05f4768be9c81ed79"
server_key_version: "encryptionkey@0"
I20260620 19:04:12.880482 21532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:12.894790 21532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:12.895411 21532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:12.895596 21532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:12.901983 21532 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:34203
I20260620 19:04:12.902055 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:34203 every 8 connection(s)
I20260620 19:04:12.902860 21532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:12.905136 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21532
I20260620 19:04:12.905256 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:12.905529 2882 external_mini_cluster.cc:1428] Setting key bf5b85f8d460b94f8fd5c7c5947d289c
I20260620 19:04:12.905970 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:12.911867 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: Bootstrap starting.
I20260620 19:04:12.913978 21599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:12.914480 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:12.906897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.2.208.190:34203)
0620 19:04:12.907573 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:12.907588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:12.907622 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:12.908483 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:12.908500 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:12.908902 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:12.909222 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:12.909627 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.910541 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.911412 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:12.911852 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:12.912015 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:12.912841 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:12.912845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:12.912859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:12.913229 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:12.913278 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:12.913297 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:12.913409 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:12.913565 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:04:12.913806 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260620 19:04:12.914712 21599 log.cc:824] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:12.916652 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: No bootstrap required, opened a new log
I20260620 19:04:12.919893 21599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } }
I20260620 19:04:12.920099 21599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:12.920132 21599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc131b9344dedaab95991e0d4600e, State: Initialized, Role: FOLLOWER
I20260620 19:04:12.920441 21599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } }
I20260620 19:04:12.920538 21599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:12.920603 21599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:12.920686 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:12.921447 21599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } }
I20260620 19:04:12.921741 21599 leader_election.cc:302] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 991cc131b9344dedaab95991e0d4600e; no voters:
I20260620 19:04:12.922045 21599 leader_election.cc:288] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:12.922144 21604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:12.922334 21604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [term 1 LEADER]: Becoming Leader. State: Replica: 991cc131b9344dedaab95991e0d4600e, State: Running, Role: LEADER
I20260620 19:04:12.922613 21604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } }
I20260620 19:04:12.923136 21599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:12.924191 21605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "991cc131b9344dedaab95991e0d4600e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } } }
I20260620 19:04:12.924324 21605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:12.924593 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 991cc131b9344dedaab95991e0d4600e. Latest consensus state: current_term: 1 leader_uuid: "991cc131b9344dedaab95991e0d4600e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991cc131b9344dedaab95991e0d4600e" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34203 } } }
I20260620 19:04:12.924686 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:12.924762 21613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:12.927263 21613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:12.932171 21613 catalog_manager.cc:1347] Generated new cluster ID: aa97ea0324c940eda476e0042dfe8ede
I20260620 19:04:12.932229 21613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:12.947965 21613 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:12.949186 21613 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:12.954664 21613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: Generated new TSK 0
I20260620 19:04:12.955310 21613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:12.966857 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--builtin_ntp_servers=127.2.208.148:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:13.075141 21623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:13.075474 21623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:13.075557 21623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:13.079643 21623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:13.079690 21623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:13.079762 21623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:13.084545 21623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:13.085624 21623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:13.086947 21623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:13.095985 21629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:13.095985 21631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:13.096005 21628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:13.096028 21623 server_base.cc:1034] running on GCE node
I20260620 19:04:13.096634 21623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:13.097214 21623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:13.098404 21623 hybrid_clock.cc:648] HybridClock initialized: now 1781982253098380 us; error 48 us; skew 500 ppm
I20260620 19:04:13.100112 21623 webserver.cc:460] Webserver started at http://127.2.208.129:34531/ using document root <none> and password file <none>
I20260620 19:04:13.100643 21623 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:13.100730 21623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:13.100975 21623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:13.102964 21623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "e13a003f948e48f0b9e936ae146a0511"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "39225c85bbb1d57df183728f5c0ff900"
server_key_iv: "8e7560f4b95c0b4031d561009f1e891a"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.103607 21623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "e13a003f948e48f0b9e936ae146a0511"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "39225c85bbb1d57df183728f5c0ff900"
server_key_iv: "8e7560f4b95c0b4031d561009f1e891a"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.106858 21623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:04:13.109133 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.110244 21623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:13.110370 21623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "e13a003f948e48f0b9e936ae146a0511"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "39225c85bbb1d57df183728f5c0ff900"
server_key_iv: "8e7560f4b95c0b4031d561009f1e891a"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.110478 21623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:13.130048 21623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:13.130650 21623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:13.130824 21623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:13.131335 21623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:13.132233 21623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:13.132299 21623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.132364 21623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:13.132408 21623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.140883 21623 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:39111
I20260620 19:04:13.140901 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:39111 every 8 connection(s)
I20260620 19:04:13.141687 21623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:13.143101 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21623
I20260620 19:04:13.143237 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:13.143621 2882 external_mini_cluster.cc:1428] Setting key 130876af919bff57dba958a57625d32a
I20260620 19:04:13.145560 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--builtin_ntp_servers=127.2.208.148:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:13.151145 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.143410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:60593 (local address 127.2.208.190:34203)
0620 19:04:13.143567 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.143572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.144704 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 19:04:13.146232 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.146238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.146320 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.146418 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:13.147531 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.148119 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.149242 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.149487 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.149598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.150662 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:13.150665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:13.150668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:13.150680 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:13.150707 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:13.150714 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:13.150784 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.150953 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.151011 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:04:13.151958 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.143844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34203 (local address 127.2.208.129:60593)
0620 19:04:13.144552 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.144604 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.146021 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.146558 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.146564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.146641 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:13.147359 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.147403 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.148240 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.148243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.149116 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.149125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.149631 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.149635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.149801 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.150427 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:13.150429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:13.150466 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:13.150522 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:13.150797 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:13.150806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:13.150811 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.150942 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.151189 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260620 19:04:13.153136 21751 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34203
I20260620 19:04:13.153404 21751 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:13.154012 21751 heartbeater.cc:511] Master 127.2.208.190:34203 requested a full tablet report, sending...
I20260620 19:04:13.155755 21561 ts_manager.cc:194] Registered new tserver with Master: e13a003f948e48f0b9e936ae146a0511 (127.2.208.129:39111)
I20260620 19:04:13.157274 21561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.129:60593
W20260620 19:04:13.253408 21754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:13.253737 21754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:13.253808 21754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:13.257702 21754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:13.257750 21754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:13.257840 21754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:13.262298 21754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:13.263345 21754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:13.264667 21754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:13.273824 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:13.273835 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:13.273862 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:13.274119 21754 server_base.cc:1034] running on GCE node
I20260620 19:04:13.274501 21754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:13.275110 21754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:13.276293 21754 hybrid_clock.cc:648] HybridClock initialized: now 1781982253276256 us; error 47 us; skew 500 ppm
I20260620 19:04:13.277953 21754 webserver.cc:460] Webserver started at http://127.2.208.130:39219/ using document root <none> and password file <none>
I20260620 19:04:13.278502 21754 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:13.278590 21754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:13.278806 21754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:13.280745 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "709ab4e402a8476f94f52523543656df"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "2f3fc6e68138720620054a9ae0a3529d"
server_key_iv: "62c6ae79ae4489d6768c2f37eff7edba"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.281299 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "709ab4e402a8476f94f52523543656df"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "2f3fc6e68138720620054a9ae0a3529d"
server_key_iv: "62c6ae79ae4489d6768c2f37eff7edba"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.284924 21754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 19:04:13.287223 21768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.288450 21754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:04:13.288575 21754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "709ab4e402a8476f94f52523543656df"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "2f3fc6e68138720620054a9ae0a3529d"
server_key_iv: "62c6ae79ae4489d6768c2f37eff7edba"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.288679 21754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:13.296804 21754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:13.297355 21754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:13.297534 21754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:13.298064 21754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:13.298899 21754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:13.298966 21754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.299028 21754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:13.299074 21754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.307502 21754 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:35075
I20260620 19:04:13.307569 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:35075 every 8 connection(s)
I20260620 19:04:13.308332 21754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:13.311628 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21754
I20260620 19:04:13.311759 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:13.312074 2882 external_mini_cluster.cc:1428] Setting key 0515ecccab12582c0a2f60b0ca8978b7
I20260620 19:04:13.314208 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--builtin_ntp_servers=127.2.208.148:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:13.319722 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.310193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:53307 (local address 127.2.208.190:34203)
0620 19:04:13.310321 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.310325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.311316 (+ 991us) server_negotiation.cc:407] Connection header received
0620 19:04:13.312888 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.312893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.312949 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.313036 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:13.313967 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.314802 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.316984 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.317430 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.317561 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.319108 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:13.319112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:13.319114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:13.319127 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:13.319149 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:13.319159 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:13.319204 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.319496 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.319562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:04:13.321081 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.310565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34203 (local address 127.2.208.130:53307)
0620 19:04:13.311181 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.311218 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.312640 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.313219 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.313227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.313333 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:13.313831 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.313844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.314955 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.314962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.316842 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.316856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.317603 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.317609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.317853 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.318676 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:13.318678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:13.318759 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:13.318888 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:13.319271 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:13.319279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:13.319314 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.319902 (+ 588us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.320140 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260620 19:04:13.324240 21882 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34203
I20260620 19:04:13.324481 21882 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:13.324981 21882 heartbeater.cc:511] Master 127.2.208.190:34203 requested a full tablet report, sending...
I20260620 19:04:13.325997 21561 ts_manager.cc:194] Registered new tserver with Master: 709ab4e402a8476f94f52523543656df (127.2.208.130:35075)
I20260620 19:04:13.326599 21561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.130:53307
W20260620 19:04:13.429591 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:13.429879 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:13.429978 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:13.434136 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:13.434207 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:13.434288 21885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:13.438969 21885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:34203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:13.440002 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:13.441430 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:13.450567 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:13.450623 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:13.450667 21885 server_base.cc:1034] running on GCE node
W20260620 19:04:13.450567 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:13.451282 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:13.451905 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:13.453100 21885 hybrid_clock.cc:648] HybridClock initialized: now 1781982253453086 us; error 24 us; skew 500 ppm
I20260620 19:04:13.454954 21885 webserver.cc:460] Webserver started at http://127.2.208.131:36575/ using document root <none> and password file <none>
I20260620 19:04:13.455579 21885 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:13.455672 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:13.455951 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:13.457919 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "93e2fdf11c8447f196997bcc0fe0b509"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "011654c6b4f5c2ad130d9f7ad18528e8"
server_key_iv: "b088aa3452ac18bce5a3ba507a18ada4"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.458402 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "93e2fdf11c8447f196997bcc0fe0b509"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "011654c6b4f5c2ad130d9f7ad18528e8"
server_key_iv: "b088aa3452ac18bce5a3ba507a18ada4"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.461854 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:13.464174 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.465305 21885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:13.465401 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "93e2fdf11c8447f196997bcc0fe0b509"
format_stamp: "Formatted at 2026-06-20 19:04:13 on dist-test-slave-tsm4"
server_key: "011654c6b4f5c2ad130d9f7ad18528e8"
server_key_iv: "b088aa3452ac18bce5a3ba507a18ada4"
server_key_version: "encryptionkey@0"
I20260620 19:04:13.465479 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:13.494256 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:13.494869 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:13.495018 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:13.495585 21885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:13.496454 21885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:13.496501 21885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.496541 21885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:13.496555 21885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:13.505183 21885 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:37859
I20260620 19:04:13.505213 22012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:37859 every 8 connection(s)
I20260620 19:04:13.506064 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:13.512173 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 21885
I20260620 19:04:13.512300 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:13.512578 2882 external_mini_cluster.cc:1428] Setting key 2b3c7eec9edfe8873927b550fbaf02c2
I20260620 19:04:13.515543 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.507772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:47557 (local address 127.2.208.190:34203)
0620 19:04:13.507913 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.507918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.509007 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 19:04:13.510306 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.510308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.510353 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.510437 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:13.511303 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.511890 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.513385 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.513644 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.513754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.515035 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:13.515038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:13.515041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:13.515054 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:13.515075 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:13.515090 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:13.515164 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.515326 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.515411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:04:13.516359 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.508218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:34203 (local address 127.2.208.131:47557)
0620 19:04:13.508858 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.508898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.510127 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.510613 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.510619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.510698 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:13.511175 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.511187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.512030 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.512034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.513264 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.513275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.513802 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.513807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.514043 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.514759 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:13.514762 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:13.514813 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:13.514892 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:13.515178 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:13.515187 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:13.515193 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.515326 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.515602 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260620 19:04:13.517407 22013 heartbeater.cc:347] Connected to a master server at 127.2.208.190:34203
I20260620 19:04:13.517583 22013 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:13.517998 22013 heartbeater.cc:511] Master 127.2.208.190:34203 requested a full tablet report, sending...
I20260620 19:04:13.518903 21561 ts_manager.cc:194] Registered new tserver with Master: 93e2fdf11c8447f196997bcc0fe0b509 (127.2.208.131:37859)
I20260620 19:04:13.519474 21561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.131:47557
I20260620 19:04:13.526365 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:13.531785 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.528303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.2.208.190:34203)
0620 19:04:13.528436 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.528440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.528601 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:04:13.528677 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.528678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.528714 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.528773 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:13.528979 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.529453 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.530242 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.530451 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.530519 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.531284 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:13.531287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:13.531289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:13.531298 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:13.531313 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:13.531319 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:13.531532 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.531657 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.531695 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:04:13.534613 21561 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:13.537626 21561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:13.551999 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.546402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37859 (local address 127.0.0.1:46516)
0620 19:04:13.546656 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.546669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.546761 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.547914 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.547916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.547934 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:13.548029 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.548036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.549001 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.549005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.550443 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.550453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551487 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.551624 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.551649 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.551739 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.551853 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":108,"thread_start_us":124,"threads_started":1}
I20260620 19:04:13.552029 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.546961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.2.208.131:37859)
0620 19:04:13.547461 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.547472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.547496 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:04:13.547564 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.547568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.547769 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.547926 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:13.548118 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.548856 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.550582 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551323 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551537 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.551722 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.551770 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.551873 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":142,"threads_started":1}
I20260620 19:04:13.552024 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.546054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39111 (local address 127.0.0.1:54790)
0620 19:04:13.546580 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.546594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.546712 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.547453 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.547460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.547487 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:13.547626 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.547632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.549093 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.549096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.550763 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.550770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551643 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.551737 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.551760 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.551820 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.551883 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":475,"thread_start_us":96,"threads_started":1}
I20260620 19:04:13.552518 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.545699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:35075 (local address 127.0.0.1:45186)
0620 19:04:13.546055 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.546118 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.546842 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.547644 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.547646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.547661 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:13.547779 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.547785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.549187 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.549189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.550256 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.550265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551327 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.551460 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.552238 (+ 778us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.552341 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.552408 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":23,"thread_start_us":102,"threads_started":1}
I20260620 19:04:13.552714 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.546511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.2.208.129:39111)
0620 19:04:13.547000 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.547011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.547032 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:04:13.547095 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.547099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.547236 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.547461 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:13.547758 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.548997 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.550880 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551518 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551636 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.551819 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.551866 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.551950 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":144,"threads_started":1}
I20260620 19:04:13.552796 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.545880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.2.208.130:35075)
0620 19:04:13.546284 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.546310 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.546338 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:04:13.547019 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.547027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.547229 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.547461 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:13.547939 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.549066 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.550385 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.551192 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.551307 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.551530 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.552384 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.552477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":115,"threads_started":1}
I20260620 19:04:13.555002 21947 tablet_service.cc:1467] Processing CreateTablet for tablet 82cf3073f0904eeea81f060a7714f144 (DEFAULT_TABLE table=test-table [id=dd441fc59080409ebc6d47544325fdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:13.555388 21685 tablet_service.cc:1467] Processing CreateTablet for tablet 82cf3073f0904eeea81f060a7714f144 (DEFAULT_TABLE table=test-table [id=dd441fc59080409ebc6d47544325fdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:13.556145 21947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cf3073f0904eeea81f060a7714f144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:13.556332 21685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cf3073f0904eeea81f060a7714f144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:13.557698 21816 tablet_service.cc:1467] Processing CreateTablet for tablet 82cf3073f0904eeea81f060a7714f144 (DEFAULT_TABLE table=test-table [id=dd441fc59080409ebc6d47544325fdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:13.558487 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cf3073f0904eeea81f060a7714f144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:13.563266 22031 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Bootstrap starting.
I20260620 19:04:13.563717 22030 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: Bootstrap starting.
I20260620 19:04:13.564602 22032 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Bootstrap starting.
I20260620 19:04:13.566337 22032 tablet_bootstrap.cc:654] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:13.566368 22031 tablet_bootstrap.cc:654] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:13.566417 22030 tablet_bootstrap.cc:654] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:13.567058 22032 log.cc:824] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:13.567265 22031 log.cc:824] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:13.567262 22030 log.cc:824] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:13.569211 22031 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: No bootstrap required, opened a new log
I20260620 19:04:13.569239 22032 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: No bootstrap required, opened a new log
I20260620 19:04:13.569393 22030 tablet_bootstrap.cc:492] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: No bootstrap required, opened a new log
I20260620 19:04:13.569427 22032 ts_tablet_manager.cc:1397] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 19:04:13.569420 22031 ts_tablet_manager.cc:1397] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:04:13.569543 22030 ts_tablet_manager.cc:1397] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:04:13.572916 22031 raft_consensus.cc:348] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.573138 22031 raft_consensus.cc:374] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:13.573190 22031 raft_consensus.cc:729] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e2fdf11c8447f196997bcc0fe0b509, State: Initialized, Role: FOLLOWER
I20260620 19:04:13.573612 22031 consensus_queue.cc:260] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.573542 22032 raft_consensus.cc:348] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.573542 22030 raft_consensus.cc:348] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.573765 22032 raft_consensus.cc:374] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:13.573765 22030 raft_consensus.cc:374] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:13.573828 22032 raft_consensus.cc:729] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709ab4e402a8476f94f52523543656df, State: Initialized, Role: FOLLOWER
I20260620 19:04:13.573822 22030 raft_consensus.cc:729] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13a003f948e48f0b9e936ae146a0511, State: Initialized, Role: FOLLOWER
I20260620 19:04:13.574266 22030 consensus_queue.cc:260] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.574266 22032 consensus_queue.cc:260] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.574638 22031 ts_tablet_manager.cc:1428] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:04:13.574954 21751 heartbeater.cc:503] Master 127.2.208.190:34203 was elected leader, sending a full tablet report...
I20260620 19:04:13.574954 22013 heartbeater.cc:503] Master 127.2.208.190:34203 was elected leader, sending a full tablet report...
I20260620 19:04:13.575544 21882 heartbeater.cc:503] Master 127.2.208.190:34203 was elected leader, sending a full tablet report...
I20260620 19:04:13.576202 22030 ts_tablet_manager.cc:1428] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:04:13.576283 22032 ts_tablet_manager.cc:1428] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260620 19:04:13.580606 22038 raft_consensus.cc:482] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:13.580713 22038 raft_consensus.cc:504] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.581533 22038 leader_election.cc:288] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709ab4e402a8476f94f52523543656df (127.2.208.130:35075), 93e2fdf11c8447f196997bcc0fe0b509 (127.2.208.131:37859)
I20260620 19:04:13.585346 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.581727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:35075 (local address 127.2.208.129:60003)
0620 19:04:13.581841 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.581854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.581940 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.582425 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.582428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.582447 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:13.582568 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.582576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.583297 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.583300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.584319 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.584328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.584975 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.584978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.585075 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.585114 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.585182 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.585259 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:04:13.585423 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.582078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:60003 (local address 127.2.208.130:35075)
0620 19:04:13.582202 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.582206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.582217 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:04:13.582261 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.582265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.582327 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.582400 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:13.582674 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.583205 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.584441 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.584887 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.584972 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.585005 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.585206 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.585270 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:13.585603 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.581939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:37859 (local address 127.2.208.129:46637)
0620 19:04:13.582243 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.582255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:13.582345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:13.582571 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:13.582573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:13.582586 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:13.582711 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.582717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.583524 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.583527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.584431 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:13.584441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.585182 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.585188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:13.585283 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.585308 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:04:13.585384 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:04:13.585446 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260620 19:04:13.585638 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.582068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:46637 (local address 127.2.208.131:37859)
0620 19:04:13.582180 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.582184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.582326 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:04:13.582436 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.582439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.582475 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.582547 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:13.582810 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.583436 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.584544 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.585042 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.585159 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.585205 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.585427 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.585517 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:13.585920 21836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cf3073f0904eeea81f060a7714f144" candidate_uuid: "e13a003f948e48f0b9e936ae146a0511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709ab4e402a8476f94f52523543656df" is_pre_election: true
I20260620 19:04:13.586108 21967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cf3073f0904eeea81f060a7714f144" candidate_uuid: "e13a003f948e48f0b9e936ae146a0511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e2fdf11c8447f196997bcc0fe0b509" is_pre_election: true
I20260620 19:04:13.586167 21836 raft_consensus.cc:2454] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13a003f948e48f0b9e936ae146a0511 in term 0.
I20260620 19:04:13.586324 21967 raft_consensus.cc:2454] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13a003f948e48f0b9e936ae146a0511 in term 0.
I20260620 19:04:13.586632 21639 leader_election.cc:302] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 709ab4e402a8476f94f52523543656df, e13a003f948e48f0b9e936ae146a0511; no voters:
I20260620 19:04:13.586880 22038 raft_consensus.cc:2781] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:13.586967 22038 raft_consensus.cc:482] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:13.587020 22038 raft_consensus.cc:3037] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:13.588021 22038 raft_consensus.cc:504] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.588331 22038 leader_election.cc:288] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [CANDIDATE]: Term 1 election: Requested vote from peers 709ab4e402a8476f94f52523543656df (127.2.208.130:35075), 93e2fdf11c8447f196997bcc0fe0b509 (127.2.208.131:37859)
I20260620 19:04:13.588639 21836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cf3073f0904eeea81f060a7714f144" candidate_uuid: "e13a003f948e48f0b9e936ae146a0511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709ab4e402a8476f94f52523543656df"
I20260620 19:04:13.588814 21967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cf3073f0904eeea81f060a7714f144" candidate_uuid: "e13a003f948e48f0b9e936ae146a0511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e2fdf11c8447f196997bcc0fe0b509"
I20260620 19:04:13.588830 21836 raft_consensus.cc:3037] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:13.588958 21967 raft_consensus.cc:3037] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:13.589819 21836 raft_consensus.cc:2454] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13a003f948e48f0b9e936ae146a0511 in term 1.
I20260620 19:04:13.589937 21967 raft_consensus.cc:2454] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13a003f948e48f0b9e936ae146a0511 in term 1.
I20260620 19:04:13.590114 21639 leader_election.cc:302] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 709ab4e402a8476f94f52523543656df, e13a003f948e48f0b9e936ae146a0511; no voters:
I20260620 19:04:13.590258 22038 raft_consensus.cc:2781] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:13.590461 22038 raft_consensus.cc:686] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 1 LEADER]: Becoming Leader. State: Replica: e13a003f948e48f0b9e936ae146a0511, State: Running, Role: LEADER
I20260620 19:04:13.590777 22038 consensus_queue.cc:237] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } }
I20260620 19:04:13.593643 21563 catalog_manager.cc:5495] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13a003f948e48f0b9e936ae146a0511 (127.2.208.129). New cstate: current_term: 1 leader_uuid: "e13a003f948e48f0b9e936ae146a0511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13a003f948e48f0b9e936ae146a0511" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 39111 } health_report { overall_health: HEALTHY } } }
I20260620 19:04:13.604970 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.601470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.2.208.129:39111)
0620 19:04:13.601586 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:13.601591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:13.601656 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:04:13.601862 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:13.601864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:13.601911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:13.602027 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:13.602264 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.602721 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.603624 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:13.603818 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:13.603903 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:13.604154 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:13.604582 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:13.604700 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:13.604836 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:04:13.604866 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:04:13.610248 21967 raft_consensus.cc:1261] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 1 FOLLOWER]: Refusing update from remote peer e13a003f948e48f0b9e936ae146a0511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:13.610248 21836 raft_consensus.cc:1261] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 1 FOLLOWER]: Refusing update from remote peer e13a003f948e48f0b9e936ae146a0511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:13.610894 22040 consensus_queue.cc:1035] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e2fdf11c8447f196997bcc0fe0b509" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:13.611048 22038 consensus_queue.cc:1035] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "709ab4e402a8476f94f52523543656df" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:13.618307 22045 mvcc.cc:205] Tried to move back new op lower bound from 7298999310780411904 to 7298999310709936128. Current Snapshot: MvccSnapshot[applied={T|T < 7298999310780411904}]
I20260620 19:04:13.618642 22044 mvcc.cc:205] Tried to move back new op lower bound from 7298999310780411904 to 7298999310709936128. Current Snapshot: MvccSnapshot[applied={T|T < 7298999310780411904}]
I20260620 19:04:13.619671 22043 mvcc.cc:205] Tried to move back new op lower bound from 7298999310780411904 to 7298999310709936128. Current Snapshot: MvccSnapshot[applied={T|T < 7298999310780411904}]
I20260620 19:04:13.627911 21561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:13.628069 21561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:13.630405 21561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 991cc131b9344dedaab95991e0d4600e: Sending DeleteTablet for 3 replicas of tablet 82cf3073f0904eeea81f060a7714f144
I20260620 19:04:13.631175 21947 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cf3073f0904eeea81f060a7714f144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:13 UTC) from {username='slave'} at 127.0.0.1:46516
I20260620 19:04:13.631175 21685 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cf3073f0904eeea81f060a7714f144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:13 UTC) from {username='slave'} at 127.0.0.1:54790
I20260620 19:04:13.631218 21816 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cf3073f0904eeea81f060a7714f144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:13 UTC) from {username='slave'} at 127.0.0.1:45186
I20260620 19:04:13.631639 22052 tablet_replica.cc:330] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511: stopping tablet replica
I20260620 19:04:13.631699 22054 tablet_replica.cc:330] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: stopping tablet replica
I20260620 19:04:13.631909 22054 raft_consensus.cc:2229] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:13.631909 22052 raft_consensus.cc:2229] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:13.632186 22054 raft_consensus.cc:2258] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:13.632239 22052 raft_consensus.cc:2258] T 82cf3073f0904eeea81f060a7714f144 P e13a003f948e48f0b9e936ae146a0511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:13.632539 22053 tablet_replica.cc:330] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: stopping tablet replica
I20260620 19:04:13.632571 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21623
I20260620 19:04:13.633420 22053 raft_consensus.cc:2229] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:13.633627 22053 raft_consensus.cc:2258] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:13.635082 22053 ts_tablet_manager.cc:1905] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:13.635490 22054 ts_tablet_manager.cc:1905] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:13.637856 22053 ts_tablet_manager.cc:1918] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:13.637970 22053 log.cc:1196] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/82cf3073f0904eeea81f060a7714f144
I20260620 19:04:13.638028 22054 ts_tablet_manager.cc:1918] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:13.638116 22054 log.cc:1196] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/82cf3073f0904eeea81f060a7714f144
I20260620 19:04:13.638329 22053 ts_tablet_manager.cc:1939] T 82cf3073f0904eeea81f060a7714f144 P 93e2fdf11c8447f196997bcc0fe0b509: Deleting consensus metadata
I20260620 19:04:13.638406 22054 ts_tablet_manager.cc:1939] T 82cf3073f0904eeea81f060a7714f144 P 709ab4e402a8476f94f52523543656df: Deleting consensus metadata
W20260620 19:04:13.639201 21547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:39111 (error 108)
I20260620 19:04:13.639452 21548 catalog_manager.cc:4841] TS 709ab4e402a8476f94f52523543656df (127.2.208.130:35075): tablet 82cf3073f0904eeea81f060a7714f144 (table test-table [id=dd441fc59080409ebc6d47544325fdd7]) successfully deleted
I20260620 19:04:13.639621 21548 catalog_manager.cc:4841] TS 93e2fdf11c8447f196997bcc0fe0b509 (127.2.208.131:37859): tablet 82cf3073f0904eeea81f060a7714f144 (table test-table [id=dd441fc59080409ebc6d47544325fdd7]) successfully deleted
I20260620 19:04:13.640013 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21754
I20260620 19:04:13.640254 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:13.639938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:39111 (local address 127.0.0.1:54800)
0620 19:04:13.640082 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:13.640160 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 19:04:13.640699 21547 catalog_manager.cc:4584] TS e13a003f948e48f0b9e936ae146a0511 (127.2.208.129:39111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82cf3073f0904eeea81f060a7714f144: Network error: Client connection negotiation failed: client connection to 127.2.208.129:39111: connect: Connection refused (error 111)
I20260620 19:04:13.646008 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21885
I20260620 19:04:13.651518 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 21532
2026-06-20T19:04:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (939 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:13 dist-test-slave-tsm4 krb5kdc[22058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:13 dist-test-slave-tsm4 krb5kdc[22058](info): set up 2 sockets
Jun 20 19:04:13 dist-test-slave-tsm4 krb5kdc[22058](info): commencing operation
krb5kdc: starting...
W20260620 19:04:14.683322 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:14 dist-test-slave-tsm4 krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:14.801656 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:42929
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:42929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:14.907994 22071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:14.908227 22071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:14.908299 22071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:14.912305 22071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:14.912353 22071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:14.912372 22071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:14.912388 22071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:14.912402 22071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:14.917524 22071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:42929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:42929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:14.918529 22071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:14.919589 22071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:14.927321 22076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:14.927359 22071 server_base.cc:1034] running on GCE node
W20260620 19:04:14.927314 22079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:14.927314 22077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:14.928154 22071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:14.929200 22071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:14.930367 22071 hybrid_clock.cc:648] HybridClock initialized: now 1781982254930354 us; error 24 us; skew 500 ppm
Jun 20 19:04:14 dist-test-slave-tsm4 krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982254, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:14.933064 22071 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:14.933899 22071 webserver.cc:460] Webserver started at http://127.2.208.190:41353/ using document root <none> and password file <none>
I20260620 19:04:14.934458 22071 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:14.934545 22071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:14.934762 22071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:14.936704 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "2921b6205b5d49bebad5dccd09e9ae55"
format_stamp: "Formatted at 2026-06-20 19:04:14 on dist-test-slave-tsm4"
server_key: "8c65c3815fa439dee68ee34cd04bfe2c"
server_key_iv: "43e628e9bc8343f29dae339f4c9ea4fb"
server_key_version: "encryptionkey@0"
I20260620 19:04:14.937248 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "2921b6205b5d49bebad5dccd09e9ae55"
format_stamp: "Formatted at 2026-06-20 19:04:14 on dist-test-slave-tsm4"
server_key: "8c65c3815fa439dee68ee34cd04bfe2c"
server_key_iv: "43e628e9bc8343f29dae339f4c9ea4fb"
server_key_version: "encryptionkey@0"
I20260620 19:04:14.940759 22071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:14.942950 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:14.944218 22071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:14.944368 22071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "2921b6205b5d49bebad5dccd09e9ae55"
format_stamp: "Formatted at 2026-06-20 19:04:14 on dist-test-slave-tsm4"
server_key: "8c65c3815fa439dee68ee34cd04bfe2c"
server_key_iv: "43e628e9bc8343f29dae339f4c9ea4fb"
server_key_version: "encryptionkey@0"
I20260620 19:04:14.944479 22071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:14.956473 22071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:14.959172 22071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:14.959429 22071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:14.966179 22071 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:42929
I20260620 19:04:14.966202 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:42929 every 8 connection(s)
I20260620 19:04:14.967173 22071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:14.969946 22139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:14.975303 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: Bootstrap starting.
I20260620 19:04:14.977087 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22071
I20260620 19:04:14.977222 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:14.977525 2882 external_mini_cluster.cc:1428] Setting key a64fe9ab758e13f4cca4c966fa61d406
I20260620 19:04:14.977859 22139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:14.978747 22139 log.cc:824] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:14.980677 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: No bootstrap required, opened a new log
I20260620 19:04:14.984758 22139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } }
I20260620 19:04:14.985028 22139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:14.985105 22139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2921b6205b5d49bebad5dccd09e9ae55, State: Initialized, Role: FOLLOWER
I20260620 19:04:14.985615 22139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } }
I20260620 19:04:14.985814 22139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:14.985893 22139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:14.985983 22139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:04:14 dist-test-slave-tsm4 krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:14.986925 22139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } }
I20260620 19:04:14.987286 22139 leader_election.cc:302] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2921b6205b5d49bebad5dccd09e9ae55; no voters:
I20260620 19:04:14.987692 22139 leader_election.cc:288] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:14.987771 22144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:14.987985 22144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [term 1 LEADER]: Becoming Leader. State: Replica: 2921b6205b5d49bebad5dccd09e9ae55, State: Running, Role: LEADER
I20260620 19:04:14.988291 22144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } }
I20260620 19:04:14.989042 22139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:14.989293 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2921b6205b5d49bebad5dccd09e9ae55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } } }
I20260620 19:04:14.989302 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2921b6205b5d49bebad5dccd09e9ae55. Latest consensus state: current_term: 1 leader_uuid: "2921b6205b5d49bebad5dccd09e9ae55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2921b6205b5d49bebad5dccd09e9ae55" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42929 } } }
I20260620 19:04:14.989423 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:14.989439 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:14.989750 22148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:14.992763 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:14.978601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.2.208.190:42929)
0620 19:04:14.979048 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:14.979063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:14.979124 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:04:14.979860 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:14.979877 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:14.980179 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:14.980447 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:14.981354 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:14.982291 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:14.983586 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:14.984035 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:14.984145 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:14.986922 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:14.986929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:14.986945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:14.986976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:14.989002 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:14.989625 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:14.989631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:14.989637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:14.989778 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:14.990137 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:14.990140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:14.990141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:14.990662 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:14.990872 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:14.991592 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 19:04:14.991866 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260620 19:04:14.993165 22148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:14.998145 22148 catalog_manager.cc:1347] Generated new cluster ID: 0afb3bf3a16b4567a4080cf887b90b5e
I20260620 19:04:14.998205 22148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:15.012406 22148 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:15.013587 22148 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:15.018139 22148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: Generated new TSK 0
I20260620 19:04:15.018752 22148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.208.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
Entry for principal kudu/127.2.208.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab.
I20260620 19:04:15.052173 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:15.165828 22165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:15.166061 22165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:15.166132 22165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:15.170199 22165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:15.170251 22165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:15.170307 22165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:15.175524 22165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.129.keytab
--principal=kudu/127.2.208.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:15.176528 22165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:15.177516 22165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:15.186582 22170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:15.186579 22173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:15.186578 22171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:15.187007 22165 server_base.cc:1034] running on GCE node
I20260620 19:04:15.187340 22165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:15.187966 22165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:15.189121 22165 hybrid_clock.cc:648] HybridClock initialized: now 1781982255189102 us; error 31 us; skew 500 ppm
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:15.192028 22165 init.cc:377] Logged in from keytab as kudu/127.2.208.129@KRBTEST.COM (short username kudu)
I20260620 19:04:15.192807 22165 webserver.cc:460] Webserver started at http://127.2.208.129:36071/ using document root <none> and password file <none>
I20260620 19:04:15.193291 22165 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:15.193346 22165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:15.193498 22165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:15.195307 22165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "431fac8bf9bb483990523aa05c369467"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "44d438dc9e63dd5637d5bc1b138f21f9"
server_key_iv: "4017fe0d06f12f14d0414e530c7a4e73"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.195845 22165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "431fac8bf9bb483990523aa05c369467"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "44d438dc9e63dd5637d5bc1b138f21f9"
server_key_iv: "4017fe0d06f12f14d0414e530c7a4e73"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.199209 22165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:15.201416 22180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.202526 22165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:15.202639 22165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "431fac8bf9bb483990523aa05c369467"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "44d438dc9e63dd5637d5bc1b138f21f9"
server_key_iv: "4017fe0d06f12f14d0414e530c7a4e73"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.202731 22165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:15.218684 22165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:15.222548 22165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:15.222746 22165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:15.223520 22165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:15.224601 22165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:15.224671 22165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.224733 22165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:15.224761 22165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.233493 22165 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:34797
I20260620 19:04:15.233522 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:34797 every 8 connection(s)
I20260620 19:04:15.234339 22165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:15.238198 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22165
I20260620 19:04:15.238336 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:15.238658 2882 external_mini_cluster.cc:1428] Setting key 6efe12f6b449f77c1dff963139a50bd3
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.129@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.130@KRBTEST.COM; defaulting to no policy
I20260620 19:04:15.249080 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.236018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:45775 (local address 127.2.208.190:42929)
0620 19:04:15.236142 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.236146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.236854 (+ 708us) server_negotiation.cc:407] Connection header received
0620 19:04:15.238247 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.238252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.238320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.238417 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:15.239739 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.240440 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.242171 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.242554 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.242670 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.245034 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:15.245040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:15.245046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:15.245080 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:15.246766 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.247336 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.247339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.247342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.247451 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.247815 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.247822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.247825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.248060 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:15.248171 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.248710 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.248860 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":44}
I20260620 19:04:15.249776 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.236309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:42929 (local address 127.2.208.129:45775)
0620 19:04:15.236717 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.236756 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.237984 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.238588 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.238594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.239012 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:15.239594 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.239610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.240574 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.240581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.242059 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.242069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.242692 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.242694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.242825 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.243466 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:15.243481 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:15.244869 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:15.246890 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.246896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.246909 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.247242 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.247581 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.247584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.247585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.247716 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.248175 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:15.248179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:15.248468 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.248623 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.248930 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.130@KRBTEST.COM" created.
I20260620 19:04:15.251128 22294 heartbeater.cc:347] Connected to a master server at 127.2.208.190:42929
I20260620 19:04:15.251436 22294 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:15.252038 22294 heartbeater.cc:511] Master 127.2.208.190:42929 requested a full tablet report, sending...
I20260620 19:04:15.253506 22103 ts_manager.cc:194] Registered new tserver with Master: 431fac8bf9bb483990523aa05c369467 (127.2.208.129:34797)
I20260620 19:04:15.254603 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.129@KRBTEST.COM'} at 127.2.208.129:45775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
Entry for principal kudu/127.2.208.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab.
I20260620 19:04:15.265949 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:15.376233 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:15.376463 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:15.376502 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:15.380448 22299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:15.380494 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:15.380548 22299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:15.385555 22299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.130.keytab
--principal=kudu/127.2.208.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:15.386448 22299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:15.387344 22299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:15.396665 22305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:15.396680 22304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:15.396668 22307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:15.396930 22299 server_base.cc:1034] running on GCE node
I20260620 19:04:15.397358 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:15.397962 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:15.399150 22299 hybrid_clock.cc:648] HybridClock initialized: now 1781982255399136 us; error 25 us; skew 500 ppm
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:15.402057 22299 init.cc:377] Logged in from keytab as kudu/127.2.208.130@KRBTEST.COM (short username kudu)
I20260620 19:04:15.402912 22299 webserver.cc:460] Webserver started at http://127.2.208.130:46311/ using document root <none> and password file <none>
I20260620 19:04:15.403545 22299 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:15.403637 22299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:15.403837 22299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:15.405792 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "90e590388ce24286a91016051f0cdb80"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "8b9f705482fdef7de8c7ddba7614c1fe"
server_key_iv: "608a296688c79fcf3dd4141a7628648f"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.406343 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "90e590388ce24286a91016051f0cdb80"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "8b9f705482fdef7de8c7ddba7614c1fe"
server_key_iv: "608a296688c79fcf3dd4141a7628648f"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.410060 22299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:04:15.412469 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.413604 22299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:15.413735 22299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "90e590388ce24286a91016051f0cdb80"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "8b9f705482fdef7de8c7ddba7614c1fe"
server_key_iv: "608a296688c79fcf3dd4141a7628648f"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.413841 22299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:15.454461 22299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:15.457211 22299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:15.457414 22299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:15.457999 22299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:15.458787 22299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:15.458858 22299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.458938 22299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:15.458981 22299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.467859 22299 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:43185
I20260620 19:04:15.467886 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:43185 every 8 connection(s)
I20260620 19:04:15.468712 22299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:15.471786 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22299
I20260620 19:04:15.471931 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:15.472278 2882 external_mini_cluster.cc:1428] Setting key a1b55a7ea8d7c557c2edf7905c3eebd4
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.130@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.208.131@KRBTEST.COM; defaulting to no policy
I20260620 19:04:15.482560 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.470419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:36273 (local address 127.2.208.190:42929)
0620 19:04:15.470558 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.470561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.471328 (+ 767us) server_negotiation.cc:407] Connection header received
0620 19:04:15.472950 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.472953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.473001 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.473104 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:15.474530 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.475034 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.476288 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.476528 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.476634 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.478933 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:15.478937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:15.478940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:15.478969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:15.480437 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.480946 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.480950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.480952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.481034 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.481379 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.481399 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.481401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.481648 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:15.481759 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.482267 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.482384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:04:15.483290 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.470723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:42929 (local address 127.2.208.130:36273)
0620 19:04:15.471171 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.471211 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.472721 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.473251 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.473257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.473729 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:15.474379 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.474401 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.475153 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.475160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.476160 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.476169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.476667 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.476671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.476826 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.477380 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:15.477394 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:15.478777 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:15.480559 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.480564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.480578 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.480844 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.481129 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.481132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.481135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.481269 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.481771 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:15.481779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:15.482043 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.482227 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.482523 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.131@KRBTEST.COM" created.
I20260620 19:04:15.484470 22428 heartbeater.cc:347] Connected to a master server at 127.2.208.190:42929
I20260620 19:04:15.484710 22428 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:15.485285 22428 heartbeater.cc:511] Master 127.2.208.190:42929 requested a full tablet report, sending...
I20260620 19:04:15.486302 22103 ts_manager.cc:194] Registered new tserver with Master: 90e590388ce24286a91016051f0cdb80 (127.2.208.130:43185)
I20260620 19:04:15.486843 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.130@KRBTEST.COM'} at 127.2.208.130:36273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
Entry for principal kudu/127.2.208.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab.
I20260620 19:04:15.498513 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:15.605199 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:15.605439 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:15.605489 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:15.609705 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:15.609762 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:15.609820 22433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:15.615217 22433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.131.keytab
--principal=kudu/127.2.208.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:15.616225 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:15.617282 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:15.627003 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:15.626979 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:15.627151 22433 server_base.cc:1034] running on GCE node
W20260620 19:04:15.627329 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:15.627792 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:15.628355 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:15.629541 22433 hybrid_clock.cc:648] HybridClock initialized: now 1781982255629525 us; error 27 us; skew 500 ppm
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:15.632681 22433 init.cc:377] Logged in from keytab as kudu/127.2.208.131@KRBTEST.COM (short username kudu)
I20260620 19:04:15.633599 22433 webserver.cc:460] Webserver started at http://127.2.208.131:45319/ using document root <none> and password file <none>
I20260620 19:04:15.634162 22433 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:15.634253 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:15.634505 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:15.636588 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "16e183e770854536b6c35b1c1ccbe4f6"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "73924c179577c5be68f543e9e9d9edf3"
server_key_iv: "be39dc7daaf4a8c20d6d542613a33b7d"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.637147 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "16e183e770854536b6c35b1c1ccbe4f6"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "73924c179577c5be68f543e9e9d9edf3"
server_key_iv: "be39dc7daaf4a8c20d6d542613a33b7d"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.640863 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:15.643500 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.644773 22433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:15.644922 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "16e183e770854536b6c35b1c1ccbe4f6"
format_stamp: "Formatted at 2026-06-20 19:04:15 on dist-test-slave-tsm4"
server_key: "73924c179577c5be68f543e9e9d9edf3"
server_key_iv: "be39dc7daaf4a8c20d6d542613a33b7d"
server_key_version: "encryptionkey@0"
I20260620 19:04:15.645035 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:15.654436 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:15.657155 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:15.657348 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:15.657905 22433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:15.658716 22433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:15.658787 22433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.658865 22433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:15.658910 22433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:15.667356 22433 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:41537
I20260620 19:04:15.667402 22561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:41537 every 8 connection(s)
I20260620 19:04:15.668249 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:15.673946 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22433
I20260620 19:04:15.674084 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:15.674397 2882 external_mini_cluster.cc:1428] Setting key 59b8663dbf5def9442df69c3c3f3c7d9
Jun 20 19:04:15 dist-test-slave-tsm4 krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982255, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.131@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:15.681277 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.669994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:39351 (local address 127.2.208.190:42929)
0620 19:04:15.670133 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.670136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.670909 (+ 773us) server_negotiation.cc:407] Connection header received
0620 19:04:15.672163 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.672165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.672209 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.672282 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:15.673379 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.674125 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.675539 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.675781 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.675909 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.678183 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:15.678186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:15.678188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:15.678212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:15.679520 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.679963 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.679965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.679967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.680037 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.680330 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.680332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.680333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.680529 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:15.680606 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.681064 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.681161 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:04:15.682063 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.670355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:42929 (local address 127.2.208.131:39351)
0620 19:04:15.670763 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.670802 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.671956 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.672435 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.672441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.672780 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:15.673228 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.673241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.674263 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.674271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.675425 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.675434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.675932 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.675935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.676086 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.676625 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:15.676641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:15.678038 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:15.679627 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.679632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.679643 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.679878 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.680112 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:15.680114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:15.680115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:15.680245 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:15.680616 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:15.680620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:15.680827 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.680982 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.681212 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260620 19:04:15.683185 22562 heartbeater.cc:347] Connected to a master server at 127.2.208.190:42929
I20260620 19:04:15.683465 22562 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:15.683933 22562 heartbeater.cc:511] Master 127.2.208.190:42929 requested a full tablet report, sending...
I20260620 19:04:15.684937 22103 ts_manager.cc:194] Registered new tserver with Master: 16e183e770854536b6c35b1c1ccbe4f6 (127.2.208.131:41537)
I20260620 19:04:15.685451 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.208.131@KRBTEST.COM'} at 127.2.208.131:39351
I20260620 19:04:15.688158 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:15.697134 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.690111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.2.208.190:42929)
0620 19:04:15.690259 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.690264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.690453 (+ 189us) server_negotiation.cc:407] Connection header received
0620 19:04:15.690546 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.690548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.690585 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.690648 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:15.691356 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.691899 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.692692 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.692895 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.692982 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.694229 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:15.694233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:15.694235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:15.694259 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:15.695508 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.695925 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.695928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.695930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.696010 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:15.696278 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:15.696282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:15.696284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:15.696499 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:15.696635 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.696856 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.696973 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:04:15.699671 22103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:15.702608 22103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:15.716576 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.710579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43185 (local address 127.0.0.1:58846)
0620 19:04:15.710880 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.710927 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.711051 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.712428 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.712433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.712477 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:15.712618 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.712625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.713761 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.713767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.715001 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.715020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.716016 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.716020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.716156 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.716197 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.716300 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.716417 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":8,"spinlock_wait_cycles":7168,"thread_start_us":68,"threads_started":1}
I20260620 19:04:15.716583 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.711601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.2.208.130:43185)
0620 19:04:15.711914 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.711925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.711946 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:04:15.712031 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.712036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.712244 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.712462 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:15.712752 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.713641 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.715132 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.715885 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.715968 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.716081 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.716350 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.716444 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":123,"threads_started":1}
I20260620 19:04:15.716766 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.711434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.2.208.131:41537)
0620 19:04:15.712449 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.712461 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.712481 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:04:15.712545 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.712550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.712718 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.712851 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:15.713028 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.714108 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.715460 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.716066 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.716175 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.716302 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.716485 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.716625 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":173,"threads_started":1}
I20260620 19:04:15.717658 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.711246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:41537 (local address 127.0.0.1:40066)
0620 19:04:15.712167 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.712179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.712258 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.712847 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.712850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.712868 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:15.712940 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.712945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.714241 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.714244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.715330 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.715337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.716174 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.716177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.716243 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.716260 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.716353 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.716422 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260620 19:04:15.718245 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.710746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34797 (local address 127.0.0.1:36416)
0620 19:04:15.711472 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.711486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.711589 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.712426 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.712432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.712473 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:15.712608 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.712614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.714238 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.714243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.715488 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.715498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.717852 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.717856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.717964 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.717985 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.718067 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.718125 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":26,"spinlock_wait_cycles":3712,"thread_start_us":76,"threads_started":1}
I20260620 19:04:15.718343 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.711280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.2.208.129:34797)
0620 19:04:15.711871 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.711887 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.711910 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:04:15.711987 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.711992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.712199 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.712754 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:15.712907 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.714084 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.716392 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.717743 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.717851 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.718004 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.718110 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.718209 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":259,"threads_started":1}
I20260620 19:04:15.719849 22362 tablet_service.cc:1467] Processing CreateTablet for tablet d897f088282745bb965073ead888719f (DEFAULT_TABLE table=test-table [id=d57ba6de64384c1eae0bab7a411859db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:15.720546 22228 tablet_service.cc:1467] Processing CreateTablet for tablet d897f088282745bb965073ead888719f (DEFAULT_TABLE table=test-table [id=d57ba6de64384c1eae0bab7a411859db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:15.720935 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897f088282745bb965073ead888719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:15.720598 22496 tablet_service.cc:1467] Processing CreateTablet for tablet d897f088282745bb965073ead888719f (DEFAULT_TABLE table=test-table [id=d57ba6de64384c1eae0bab7a411859db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:15.721282 22228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897f088282745bb965073ead888719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:15.721509 22496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897f088282745bb965073ead888719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:15.727174 22579 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: Bootstrap starting.
I20260620 19:04:15.727878 22581 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Bootstrap starting.
I20260620 19:04:15.728392 22580 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Bootstrap starting.
I20260620 19:04:15.730088 22580 tablet_bootstrap.cc:654] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:15.730566 22579 tablet_bootstrap.cc:654] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:15.730608 22581 tablet_bootstrap.cc:654] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:15.731123 22580 log.cc:824] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:15.731458 22581 log.cc:824] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:15.731698 22579 log.cc:824] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:15.732990 22580 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: No bootstrap required, opened a new log
I20260620 19:04:15.733197 22580 ts_tablet_manager.cc:1397] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:04:15.733368 22581 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: No bootstrap required, opened a new log
I20260620 19:04:15.733368 22579 tablet_bootstrap.cc:492] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: No bootstrap required, opened a new log
I20260620 19:04:15.733542 22581 ts_tablet_manager.cc:1397] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:04:15.733542 22579 ts_tablet_manager.cc:1397] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:04:15.736196 22580 raft_consensus.cc:348] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.736421 22580 raft_consensus.cc:374] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:15.736478 22580 raft_consensus.cc:729] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e183e770854536b6c35b1c1ccbe4f6, State: Initialized, Role: FOLLOWER
I20260620 19:04:15.736889 22580 consensus_queue.cc:260] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.737152 22579 raft_consensus.cc:348] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.737392 22579 raft_consensus.cc:374] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:15.737450 22579 raft_consensus.cc:729] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431fac8bf9bb483990523aa05c369467, State: Initialized, Role: FOLLOWER
I20260620 19:04:15.737779 22562 heartbeater.cc:503] Master 127.2.208.190:42929 was elected leader, sending a full tablet report...
I20260620 19:04:15.737895 22580 ts_tablet_manager.cc:1428] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 19:04:15.737847 22579 consensus_queue.cc:260] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.738411 22294 heartbeater.cc:503] Master 127.2.208.190:42929 was elected leader, sending a full tablet report...
I20260620 19:04:15.737152 22581 raft_consensus.cc:348] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.738723 22579 ts_tablet_manager.cc:1428] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:04:15.738720 22581 raft_consensus.cc:374] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:15.738776 22581 raft_consensus.cc:729] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e590388ce24286a91016051f0cdb80, State: Initialized, Role: FOLLOWER
I20260620 19:04:15.739177 22581 consensus_queue.cc:260] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.739780 22428 heartbeater.cc:503] Master 127.2.208.190:42929 was elected leader, sending a full tablet report...
I20260620 19:04:15.740356 22581 ts_tablet_manager.cc:1428] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260620 19:04:15.880540 22585 raft_consensus.cc:482] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:15.880753 22585 raft_consensus.cc:504] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.881953 22585 leader_election.cc:288] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90e590388ce24286a91016051f0cdb80 (127.2.208.130:43185), 431fac8bf9bb483990523aa05c369467 (127.2.208.129:34797)
I20260620 19:04:15.885732 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.882177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34797 (local address 127.2.208.131:51713)
0620 19:04:15.882285 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.882300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.882389 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.882868 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.882871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.882891 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:15.883046 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.883054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.883914 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.883917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.884716 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.884724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.885340 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.885346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.885442 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.885470 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.885539 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.885590 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:04:15.885792 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.882447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:51713 (local address 127.2.208.129:34797)
0620 19:04:15.882600 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.882605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.882620 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:04:15.882683 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.882689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.882744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.882846 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:15.883162 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.883805 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.884844 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.885229 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.885318 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.885397 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.885577 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.885664 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:04:15.885854 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.882294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:41681 (local address 127.2.208.130:43185)
0620 19:04:15.882422 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.882426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.882742 (+ 316us) server_negotiation.cc:407] Connection header received
0620 19:04:15.882870 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.882873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.882920 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.883007 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:15.883288 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.883817 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.884867 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.885262 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.885391 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.885441 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.885665 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.885727 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:04:15.885865 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.882183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43185 (local address 127.2.208.131:41681)
0620 19:04:15.882640 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.882653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:15.882752 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:15.883035 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:15.883037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:15.883050 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:15.883177 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.883183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.883950 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.883953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.884724 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:15.884732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.885413 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.885418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:15.885516 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.885561 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:04:15.885634 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:04:15.885693 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":28,"thread_start_us":113,"threads_started":1}
I20260620 19:04:15.886395 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897f088282745bb965073ead888719f" candidate_uuid: "16e183e770854536b6c35b1c1ccbe4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431fac8bf9bb483990523aa05c369467" is_pre_election: true
I20260620 19:04:15.886469 22382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897f088282745bb965073ead888719f" candidate_uuid: "16e183e770854536b6c35b1c1ccbe4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e590388ce24286a91016051f0cdb80" is_pre_election: true
I20260620 19:04:15.886705 22248 raft_consensus.cc:2454] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16e183e770854536b6c35b1c1ccbe4f6 in term 0.
I20260620 19:04:15.886787 22382 raft_consensus.cc:2454] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16e183e770854536b6c35b1c1ccbe4f6 in term 0.
I20260620 19:04:15.887176 22450 leader_election.cc:302] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e183e770854536b6c35b1c1ccbe4f6, 431fac8bf9bb483990523aa05c369467; no voters:
I20260620 19:04:15.887444 22585 raft_consensus.cc:2781] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:15.887527 22585 raft_consensus.cc:482] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:15.887574 22585 raft_consensus.cc:3037] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:15.888617 22585 raft_consensus.cc:504] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.889020 22585 leader_election.cc:288] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [CANDIDATE]: Term 1 election: Requested vote from peers 90e590388ce24286a91016051f0cdb80 (127.2.208.130:43185), 431fac8bf9bb483990523aa05c369467 (127.2.208.129:34797)
I20260620 19:04:15.889396 22248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897f088282745bb965073ead888719f" candidate_uuid: "16e183e770854536b6c35b1c1ccbe4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431fac8bf9bb483990523aa05c369467"
I20260620 19:04:15.889427 22382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897f088282745bb965073ead888719f" candidate_uuid: "16e183e770854536b6c35b1c1ccbe4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e590388ce24286a91016051f0cdb80"
I20260620 19:04:15.889544 22248 raft_consensus.cc:3037] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:15.889550 22382 raft_consensus.cc:3037] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:15.890646 22248 raft_consensus.cc:2454] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16e183e770854536b6c35b1c1ccbe4f6 in term 1.
I20260620 19:04:15.890741 22382 raft_consensus.cc:2454] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16e183e770854536b6c35b1c1ccbe4f6 in term 1.
I20260620 19:04:15.890949 22450 leader_election.cc:302] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e183e770854536b6c35b1c1ccbe4f6, 431fac8bf9bb483990523aa05c369467; no voters:
I20260620 19:04:15.891104 22585 raft_consensus.cc:2781] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:15.891304 22585 raft_consensus.cc:686] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 1 LEADER]: Becoming Leader. State: Replica: 16e183e770854536b6c35b1c1ccbe4f6, State: Running, Role: LEADER
I20260620 19:04:15.891633 22585 consensus_queue.cc:237] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } }
I20260620 19:04:15.894346 22103 catalog_manager.cc:5495] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16e183e770854536b6c35b1c1ccbe4f6 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "16e183e770854536b6c35b1c1ccbe4f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16e183e770854536b6c35b1c1ccbe4f6" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 41537 } health_report { overall_health: HEALTHY } } }
I20260620 19:04:15.908226 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.904639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.2.208.131:41537)
0620 19:04:15.904756 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:15.904763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:15.904777 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:15.904957 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:15.904959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:15.905005 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:15.905107 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:15.905330 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.905818 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.906675 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:15.906889 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:15.907018 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:15.907352 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:15.907850 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:15.907981 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:15.908044 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:04:15.908095 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:04:15.913681 22248 raft_consensus.cc:1261] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 1 FOLLOWER]: Refusing update from remote peer 16e183e770854536b6c35b1c1ccbe4f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:15.913681 22382 raft_consensus.cc:1261] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 1 FOLLOWER]: Refusing update from remote peer 16e183e770854536b6c35b1c1ccbe4f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:15.914336 22589 consensus_queue.cc:1035] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e590388ce24286a91016051f0cdb80" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:15.914474 22585 consensus_queue.cc:1035] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431fac8bf9bb483990523aa05c369467" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:04:15.919121 22563 tablet.cc:2369] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:15.923213 22594 mvcc.cc:205] Tried to move back new op lower bound from 7298999320216141824 to 7298999320134873088. Current Snapshot: MvccSnapshot[applied={T|T < 7298999320216141824}]
I20260620 19:04:15.929800 22102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:15.929987 22102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:15.932370 22102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2921b6205b5d49bebad5dccd09e9ae55: Sending DeleteTablet for 3 replicas of tablet d897f088282745bb965073ead888719f
I20260620 19:04:15.933109 22228 tablet_service.cc:1514] Processing DeleteTablet for tablet d897f088282745bb965073ead888719f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:36416
I20260620 19:04:15.933176 22362 tablet_service.cc:1514] Processing DeleteTablet for tablet d897f088282745bb965073ead888719f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:58846
I20260620 19:04:15.933153 22496 tablet_service.cc:1514] Processing DeleteTablet for tablet d897f088282745bb965073ead888719f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:15 UTC) from {username='kudu', principal='kudu/127.2.208.190@KRBTEST.COM'} at 127.0.0.1:40066
I20260620 19:04:15.933601 22604 tablet_replica.cc:330] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: stopping tablet replica
I20260620 19:04:15.933823 22602 tablet_replica.cc:330] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467: stopping tablet replica
I20260620 19:04:15.933892 22604 raft_consensus.cc:2229] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:15.934162 22602 raft_consensus.cc:2229] T d897f088282745bb965073ead888719f P 431fac8bf9bb483990523aa05c369467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:15.934237 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22165
I20260620 19:04:15.934254 22604 raft_consensus.cc:2258] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:15.936761 22604 ts_tablet_manager.cc:1905] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:15.939592 22603 tablet_replica.cc:330] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: stopping tablet replica
I20260620 19:04:15.939972 22603 raft_consensus.cc:2229] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:15.940299 22604 ts_tablet_manager.cc:1918] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:15.940327 22603 raft_consensus.cc:2258] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:15.940400 22604 log.cc:1196] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/d897f088282745bb965073ead888719f
I20260620 19:04:15.941112 22604 ts_tablet_manager.cc:1939] T d897f088282745bb965073ead888719f P 16e183e770854536b6c35b1c1ccbe4f6: Deleting consensus metadata
I20260620 19:04:15.941895 22603 ts_tablet_manager.cc:1905] T d897f088282745bb965073ead888719f P 90e590388ce24286a91016051f0cdb80: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:04:15.942303 22088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:34797 (error 108)
I20260620 19:04:15.942919 22087 catalog_manager.cc:4841] TS 16e183e770854536b6c35b1c1ccbe4f6 (127.2.208.131:41537): tablet d897f088282745bb965073ead888719f (table test-table [id=d57ba6de64384c1eae0bab7a411859db]) successfully deleted
W20260620 19:04:15.943639 22574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:15.943272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:34797 (local address 127.0.0.1:36426)
0620 19:04:15.943438 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.943521 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34797: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:04:15.943804 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22299
W20260620 19:04:15.943895 22088 catalog_manager.cc:4584] TS 431fac8bf9bb483990523aa05c369467 (127.2.208.129:34797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d897f088282745bb965073ead888719f: Network error: Client connection negotiation failed: client connection to 127.2.208.129:34797: connect: Connection reset by peer (error 104)
I20260620 19:04:15.950311 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22433
I20260620 19:04:15.950886 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:15.950564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:43185 (local address 127.0.0.1:58862)
0620 19:04:15.950742 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:15.950794 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.130:43185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:04:15.955888 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22071
2026-06-20T19:04:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:15Z Disabled control of system clock
I20260620 19:04:15.985435 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41451
--webserver_interface=127.2.208.190
--webserver_port=0
--builtin_ntp_servers=127.2.208.148:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:16.088900 22609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:16.089125 22609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:16.089159 22609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:16.093809 22609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:16.093891 22609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:16.093941 22609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:16.093961 22609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:16.093976 22609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:16.098894 22609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:41451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:16.099937 22609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:16.101205 22609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:16.109586 22614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:16.109547 22615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.109673 22609 server_base.cc:1034] running on GCE node
W20260620 19:04:16.109549 22617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.110317 22609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:16.111482 22609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:16.112665 22609 hybrid_clock.cc:648] HybridClock initialized: now 1781982256112645 us; error 31 us; skew 500 ppm
I20260620 19:04:16.114382 22609 webserver.cc:460] Webserver started at http://127.2.208.190:43913/ using document root <none> and password file <none>
I20260620 19:04:16.114926 22609 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:16.115016 22609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:16.115257 22609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:16.117267 22609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "9c2848b10b76489b8a383d05651e8bbf"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "837dd46073ae8a1e44540811e2e9614d"
server_key_iv: "67bcc27820786b9a488aeecdbf37aa21"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.117842 22609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "9c2848b10b76489b8a383d05651e8bbf"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "837dd46073ae8a1e44540811e2e9614d"
server_key_iv: "67bcc27820786b9a488aeecdbf37aa21"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.121330 22609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:16.123523 22623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.124629 22609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:16.124825 22609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9c2848b10b76489b8a383d05651e8bbf"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "837dd46073ae8a1e44540811e2e9614d"
server_key_iv: "67bcc27820786b9a488aeecdbf37aa21"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.124950 22609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:16.135488 22609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:16.136080 22609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:16.136288 22609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:16.142854 22675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:41451 every 8 connection(s)
I20260620 19:04:16.142854 22609 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:41451
I20260620 19:04:16.143836 22609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:16.146754 22676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:16.150311 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22609
I20260620 19:04:16.150435 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:16.150755 2882 external_mini_cluster.cc:1428] Setting key a957fe4a5984a0346e7e223bc8c34b67
I20260620 19:04:16.152308 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: Bootstrap starting.
I20260620 19:04:16.154778 22676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:16.155625 22676 log.cc:824] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:16.157414 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: No bootstrap required, opened a new log
I20260620 19:04:16.160647 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.152192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.2.208.190:41451)
0620 19:04:16.152816 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.152834 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.152872 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:04:16.153666 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.153688 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.154143 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.154474 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:16.154910 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.156262 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.157109 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.157788 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.157950 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.158704 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:16.158710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:16.158725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:16.159195 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:16.159255 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:16.159281 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:16.159402 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.159536 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.159777 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260620 19:04:16.161804 22676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } }
I20260620 19:04:16.162041 22676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:16.162099 22676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2848b10b76489b8a383d05651e8bbf, State: Initialized, Role: FOLLOWER
I20260620 19:04:16.162516 22676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } }
I20260620 19:04:16.162675 22676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:16.162760 22676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:16.162845 22676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:16.163678 22676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } }
I20260620 19:04:16.163990 22676 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2848b10b76489b8a383d05651e8bbf; no voters:
I20260620 19:04:16.164243 22676 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:16.164407 22681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:16.164636 22681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [term 1 LEADER]: Becoming Leader. State: Replica: 9c2848b10b76489b8a383d05651e8bbf, State: Running, Role: LEADER
I20260620 19:04:16.164899 22681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } }
I20260620 19:04:16.165294 22676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:16.166527 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2848b10b76489b8a383d05651e8bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } } }
I20260620 19:04:16.166482 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2848b10b76489b8a383d05651e8bbf. Latest consensus state: current_term: 1 leader_uuid: "9c2848b10b76489b8a383d05651e8bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2848b10b76489b8a383d05651e8bbf" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41451 } } }
I20260620 19:04:16.166615 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:16.166615 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:16.167028 22690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:16.169803 22690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:16.174816 22690 catalog_manager.cc:1347] Generated new cluster ID: a180306df81a40c29a4c4dbd67c16269
I20260620 19:04:16.174875 22690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:16.201375 22690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:16.202414 22690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:16.207820 22690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: Generated new TSK 0
I20260620 19:04:16.208472 22690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:16.215631 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.129:0
--local_ip_for_outbound_sockets=127.2.208.129
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--builtin_ntp_servers=127.2.208.148:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:04:16.324283 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:16.324517 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:16.324553 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:16.328552 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:16.328599 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:16.328683 22700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.129
I20260620 19:04:16.333520 22700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.208.129
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.129
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:16.334508 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:16.335886 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:16.345082 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:16.345088 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.345242 22700 server_base.cc:1034] running on GCE node
W20260620 19:04:16.345364 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.345820 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:16.346644 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:16.347806 22700 hybrid_clock.cc:648] HybridClock initialized: now 1781982256347752 us; error 95 us; skew 500 ppm
I20260620 19:04:16.349619 22700 webserver.cc:460] Webserver started at http://127.2.208.129:34379/ using document root <none> and password file <none>
I20260620 19:04:16.350144 22700 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:16.350200 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:16.350353 22700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:16.352478 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data/instance:
uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "a653378d87744205d3c16b3471566d47"
server_key_iv: "97bcdfadf93cdee56317f972f5461963"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.352972 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance:
uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "a653378d87744205d3c16b3471566d47"
server_key_iv: "97bcdfadf93cdee56317f972f5461963"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.356426 22700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:16.358626 22714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.359876 22700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:16.360018 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal
uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "a653378d87744205d3c16b3471566d47"
server_key_iv: "97bcdfadf93cdee56317f972f5461963"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.360114 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:16.374541 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:16.375262 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:16.375464 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:16.376006 22700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:16.376870 22700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:16.376920 22700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.376993 22700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:16.377022 22700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.385543 22700 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.129:33765
I20260620 19:04:16.385563 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.129:33765 every 8 connection(s)
I20260620 19:04:16.386425 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/data/info.pb
I20260620 19:04:16.391526 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22700
I20260620 19:04:16.391649 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-0/wal/instance
I20260620 19:04:16.391965 2882 external_mini_cluster.cc:1428] Setting key 8c791da7ad5e682ff9eb411e5b7c476d
I20260620 19:04:16.393724 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.130:0
--local_ip_for_outbound_sockets=127.2.208.130
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--builtin_ntp_servers=127.2.208.148:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:16.395748 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.388310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.129:52179 (local address 127.2.208.190:41451)
0620 19:04:16.388440 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.388444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.389454 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 19:04:16.390812 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.390815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.390860 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.390942 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:16.391950 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.392609 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.393708 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.394020 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.394130 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.395247 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:16.395251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:16.395253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:16.395285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:16.395301 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:16.395308 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:16.395401 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.395564 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.395620 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:04:16.396582 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.388706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41451 (local address 127.2.208.129:52179)
0620 19:04:16.389312 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.389350 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.390578 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.391110 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.391116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.391198 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:16.391812 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.391826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.392740 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.392743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.393581 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.393591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.394164 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.394168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.394337 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.394956 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:16.394958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:16.394994 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:16.395081 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:16.395405 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:16.395416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:16.395421 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.395563 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.395814 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260620 19:04:16.397780 22828 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41451
I20260620 19:04:16.398025 22828 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:16.398643 22828 heartbeater.cc:511] Master 127.2.208.190:41451 requested a full tablet report, sending...
I20260620 19:04:16.400388 22640 ts_manager.cc:194] Registered new tserver with Master: c5ceb0f0f8bf4de0aa665d58e4ebf345 (127.2.208.129:33765)
I20260620 19:04:16.402123 22640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.129:52179
W20260620 19:04:16.501896 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:16.502115 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:16.502149 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:16.506013 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:16.506074 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:16.506147 22831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.130
I20260620 19:04:16.510995 22831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.208.130
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.130
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:16.511921 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:16.513079 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:16.521940 22836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:16.521966 22837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.522162 22831 server_base.cc:1034] running on GCE node
W20260620 19:04:16.522301 22839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.522711 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:16.523249 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:16.524401 22831 hybrid_clock.cc:648] HybridClock initialized: now 1781982256524390 us; error 23 us; skew 500 ppm
I20260620 19:04:16.526029 22831 webserver.cc:460] Webserver started at http://127.2.208.130:46535/ using document root <none> and password file <none>
I20260620 19:04:16.526580 22831 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:16.526639 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:16.526880 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:16.528935 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data/instance:
uuid: "72da029db4b9445a854367c00a2f1263"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "b5f016f87e7865dac0aac2f6cb9101f8"
server_key_iv: "75c81556cf69b8d92d9e3fd826c62627"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.529464 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance:
uuid: "72da029db4b9445a854367c00a2f1263"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "b5f016f87e7865dac0aac2f6cb9101f8"
server_key_iv: "75c81556cf69b8d92d9e3fd826c62627"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.532796 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 19:04:16.535017 22845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.536152 22831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:16.536322 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal
uuid: "72da029db4b9445a854367c00a2f1263"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "b5f016f87e7865dac0aac2f6cb9101f8"
server_key_iv: "75c81556cf69b8d92d9e3fd826c62627"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.536423 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:16.560820 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:16.561416 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:16.561599 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:16.562134 22831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:16.562987 22831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:16.563052 22831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.563118 22831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:16.563160 22831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.571928 22831 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.130:42943
I20260620 19:04:16.571950 22958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.130:42943 every 8 connection(s)
I20260620 19:04:16.572746 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/data/info.pb
I20260620 19:04:16.579883 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22831
I20260620 19:04:16.580024 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal/instance
I20260620 19:04:16.580353 2882 external_mini_cluster.cc:1428] Setting key 9fda3cd254524ff0ea80e8dce1bb2bd2
I20260620 19:04:16.581535 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.574369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.130:58803 (local address 127.2.208.190:41451)
0620 19:04:16.574523 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.574528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.575617 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 19:04:16.576961 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.576963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.577015 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.577106 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:16.577977 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.578496 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.579593 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.579812 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.579889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.581069 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:16.581072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:16.581074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:16.581085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:16.581100 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:16.581107 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:16.581192 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.581345 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.581401 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:04:16.582290 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.208.131:0
--local_ip_for_outbound_sockets=127.2.208.131
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--builtin_ntp_servers=127.2.208.148:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:04:16.582314 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.574760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41451 (local address 127.2.208.130:58803)
0620 19:04:16.575454 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.575502 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.576757 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.577280 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.577287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.577385 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:16.577852 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.577863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.578607 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.578609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.579491 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.579500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.579971 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.579977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.580193 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.580790 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:16.580791 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:16.580829 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:16.580906 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:16.581193 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:16.581203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:16.581211 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.581343 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.581620 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260620 19:04:16.583420 22959 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41451
I20260620 19:04:16.583680 22959 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:16.584179 22959 heartbeater.cc:511] Master 127.2.208.190:41451 requested a full tablet report, sending...
I20260620 19:04:16.585173 22640 ts_manager.cc:194] Registered new tserver with Master: 72da029db4b9445a854367c00a2f1263 (127.2.208.130:42943)
I20260620 19:04:16.585958 22640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.130:58803
W20260620 19:04:16.689230 22962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:16.689463 22962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:16.689499 22962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:16.693388 22962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:16.693436 22962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:16.693486 22962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.208.131
I20260620 19:04:16.697911 22962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.208.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.208.131
--webserver_port=0
--tserver_master_addrs=127.2.208.190:41451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.208.131
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:16.698843 22962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:16.700146 22962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:16.709054 22967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:16.709053 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:16.709260 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:16.709283 22962 server_base.cc:1034] running on GCE node
I20260620 19:04:16.709740 22962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:16.710404 22962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:16.711557 22962 hybrid_clock.cc:648] HybridClock initialized: now 1781982256711545 us; error 23 us; skew 500 ppm
I20260620 19:04:16.713138 22962 webserver.cc:460] Webserver started at http://127.2.208.131:39219/ using document root <none> and password file <none>
I20260620 19:04:16.713616 22962 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:16.713670 22962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:16.713819 22962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:16.715713 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data/instance:
uuid: "703ba0cc10a547c398474c787d257d78"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "9c6617682b758fff74f5de0f8f0ef921"
server_key_iv: "a233ec8fbf0afa6987e1fe44012d029a"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.716185 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance:
uuid: "703ba0cc10a547c398474c787d257d78"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "9c6617682b758fff74f5de0f8f0ef921"
server_key_iv: "a233ec8fbf0afa6987e1fe44012d029a"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.719555 22962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:16.721704 22976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.722858 22962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:16.722985 22962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal
uuid: "703ba0cc10a547c398474c787d257d78"
format_stamp: "Formatted at 2026-06-20 19:04:16 on dist-test-slave-tsm4"
server_key: "9c6617682b758fff74f5de0f8f0ef921"
server_key_iv: "a233ec8fbf0afa6987e1fe44012d029a"
server_key_version: "encryptionkey@0"
I20260620 19:04:16.723083 22962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:16.734076 22962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:16.734658 22962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:16.734833 22962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:16.735340 22962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:04:16.736225 22962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:04:16.736266 22962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.736325 22962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:04:16.736359 22962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:16.744760 22962 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.131:39005
I20260620 19:04:16.744787 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.131:39005 every 8 connection(s)
I20260620 19:04:16.745568 22962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/data/info.pb
I20260620 19:04:16.748437 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 22962
I20260620 19:04:16.748557 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal/instance
I20260620 19:04:16.748836 2882 external_mini_cluster.cc:1428] Setting key b64c3d42015fa5d55edff425a524d30b
I20260620 19:04:16.754330 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.747203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:49909 (local address 127.2.208.190:41451)
0620 19:04:16.747328 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.747332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.748341 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 19:04:16.749917 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.749921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.749975 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.750065 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:16.750988 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.751561 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.752618 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.752836 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.752980 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.753895 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:16.753898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:16.753901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:16.753918 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:16.753937 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:16.753944 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:16.754012 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.754154 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.754207 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:04:16.755066 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.747598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:41451 (local address 127.2.208.131:49909)
0620 19:04:16.748196 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.748233 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.749720 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.750244 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.750250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.750327 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:16.750850 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.750865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.751674 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.751677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.752497 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.752506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.753011 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.753015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.753162 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.753664 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:04:16.753665 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:16.753701 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:04:16.753764 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:16.754025 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:16.754034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:16.754045 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.754153 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.754384 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260620 19:04:16.756029 23090 heartbeater.cc:347] Connected to a master server at 127.2.208.190:41451
I20260620 19:04:16.756237 23090 heartbeater.cc:464] Registering TS with master...
I20260620 19:04:16.756699 23090 heartbeater.cc:511] Master 127.2.208.190:41451 requested a full tablet report, sending...
I20260620 19:04:16.757609 22640 ts_manager.cc:194] Registered new tserver with Master: 703ba0cc10a547c398474c787d257d78 (127.2.208.131:39005)
I20260620 19:04:16.758162 22640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.208.131:49909
I20260620 19:04:16.762347 2882 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:04:16.767841 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.764184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.2.208.190:41451)
0620 19:04:16.764302 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.764307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.764498 (+ 191us) server_negotiation.cc:407] Connection header received
0620 19:04:16.764571 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.764573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.764608 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.764662 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:16.764875 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.765411 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.766358 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.766561 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.766631 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.767459 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:16.767461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:16.767463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:16.767473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:16.767487 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:16.767494 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:16.767557 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.767603 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.767643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:16.770469 22640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:04:16.773406 22640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:04:16.787161 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.781294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.131:39005 (local address 127.0.0.1:49834)
0620 19:04:16.782216 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.782228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.782298 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.783014 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.783019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.783042 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:16.783182 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.783190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.784180 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.784184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.785477 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.785486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.786552 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.786556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.786667 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.786774 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.786870 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.787005 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":262,"thread_start_us":110,"threads_started":1}
I20260620 19:04:16.787192 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.781971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.2.208.131:39005)
0620 19:04:16.782542 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.782554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.782576 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:04:16.782648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.782653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.782843 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.782997 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:16.783294 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.784068 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.785602 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.786422 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.786555 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.786736 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.786915 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.787005 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":140,"threads_started":1}
I20260620 19:04:16.787190 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.781235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33765 (local address 127.0.0.1:51428)
0620 19:04:16.781826 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.781880 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.782030 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.783149 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.783152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.783182 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:16.783321 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.783329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.784286 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.784290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.785612 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.785622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.786578 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.786581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.786696 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.786755 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.786835 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.786999 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":31,"spinlock_wait_cycles":3840,"thread_start_us":73,"threads_started":1}
I20260620 19:04:16.787791 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.781344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.2.208.129:33765)
0620 19:04:16.782739 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.782747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.782764 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:04:16.782820 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.782823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.782999 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.783179 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:16.783450 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.784166 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.785727 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.786453 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.787295 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.787493 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.787540 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.787622 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":109,"threads_started":1}
I20260620 19:04:16.789034 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.781626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:42943 (local address 127.0.0.1:56036)
0620 19:04:16.782424 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.782436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.782536 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.783149 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.783152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.783180 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:16.784546 (+ 1366us) spinlock_profiling.cc:243] Waited 932 us on lock 0x2454c968. stack: 00007fdb28126c78 00007fdb281271a9 00007fdb27ce91d7 00000000005984a6 00000000005984f9 000000000059c488 00007fdb281232ab 00007fdb28122ad0 00007fdb30efa0a5 00007fdb30efa58c 00007fdb28ba41fc 00007fdb2c779b1f 00007fdb2c7ba832 00007fdb2c7bb68c 00007fdb2c7d10d3 00007fdb2c7d454b
0620 19:04:16.784609 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.784615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.785958 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.785963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.786788 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.786796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.788643 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.788647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.788728 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.788748 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.788807 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.788892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":21,"spinlock_wait_cycles":2149888,"thread_start_us":79,"threads_started":1}
I20260620 19:04:16.789130 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.782103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.2.208.130:42943)
0620 19:04:16.782542 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.782554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.782576 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:04:16.782648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.782653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.782843 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.782997 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:16.784777 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.785809 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.787344 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.788534 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.788655 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.788823 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.788884 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.788966 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":88,"threads_started":1}
I20260620 19:04:16.790400 23024 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc (DEFAULT_TABLE table=test-table [id=5ed982175dd342ef8c206512527e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:16.790644 22762 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc (DEFAULT_TABLE table=test-table [id=5ed982175dd342ef8c206512527e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:16.791560 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb06524ebb546b6aaa06d033a2db3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:16.791560 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb06524ebb546b6aaa06d033a2db3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:16.791213 22893 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc (DEFAULT_TABLE table=test-table [id=5ed982175dd342ef8c206512527e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:04:16.792202 22893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb06524ebb546b6aaa06d033a2db3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:16.797763 23107 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Bootstrap starting.
I20260620 19:04:16.798857 23108 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Bootstrap starting.
I20260620 19:04:16.799639 23109 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Bootstrap starting.
I20260620 19:04:16.800184 23107 tablet_bootstrap.cc:654] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:16.800841 23108 tablet_bootstrap.cc:654] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:16.800997 23107 log.cc:824] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:16.801314 23109 tablet_bootstrap.cc:654] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:16.801591 23108 log.cc:824] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:16.802111 23109 log.cc:824] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:16.803078 23107 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: No bootstrap required, opened a new log
I20260620 19:04:16.803282 23107 ts_tablet_manager.cc:1397] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:04:16.803764 23109 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: No bootstrap required, opened a new log
I20260620 19:04:16.803861 23108 tablet_bootstrap.cc:492] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: No bootstrap required, opened a new log
I20260620 19:04:16.803959 23109 ts_tablet_manager.cc:1397] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:04:16.804032 23108 ts_tablet_manager.cc:1397] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:04:16.806726 23109 raft_consensus.cc:348] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.806941 23109 raft_consensus.cc:374] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:16.806990 23109 raft_consensus.cc:729] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703ba0cc10a547c398474c787d257d78, State: Initialized, Role: FOLLOWER
I20260620 19:04:16.807448 23109 consensus_queue.cc:260] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.807391 23107 raft_consensus.cc:348] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.807623 23107 raft_consensus.cc:374] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:16.807685 23107 raft_consensus.cc:729] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da029db4b9445a854367c00a2f1263, State: Initialized, Role: FOLLOWER
I20260620 19:04:16.807885 23108 raft_consensus.cc:348] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.808104 23108 raft_consensus.cc:374] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:16.808157 23108 raft_consensus.cc:729] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ceb0f0f8bf4de0aa665d58e4ebf345, State: Initialized, Role: FOLLOWER
I20260620 19:04:16.808102 23107 consensus_queue.cc:260] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.808543 23108 consensus_queue.cc:260] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.808470 23109 ts_tablet_manager.cc:1428] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 19:04:16.808954 23090 heartbeater.cc:503] Master 127.2.208.190:41451 was elected leader, sending a full tablet report...
I20260620 19:04:16.809012 23107 ts_tablet_manager.cc:1428] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:04:16.809334 22959 heartbeater.cc:503] Master 127.2.208.190:41451 was elected leader, sending a full tablet report...
I20260620 19:04:16.809384 23108 ts_tablet_manager.cc:1428] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:04:16.809697 22828 heartbeater.cc:503] Master 127.2.208.190:41451 was elected leader, sending a full tablet report...
W20260620 19:04:16.823648 22960 tablet.cc:2369] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:04:16.887658 22829 tablet.cc:2369] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:16.894948 23113 raft_consensus.cc:482] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:16.895164 23113 raft_consensus.cc:504] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.896334 23113 leader_election.cc:288] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ceb0f0f8bf4de0aa665d58e4ebf345 (127.2.208.129:33765), 72da029db4b9445a854367c00a2f1263 (127.2.208.130:42943)
I20260620 19:04:16.900158 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.896709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:59867 (local address 127.2.208.130:42943)
0620 19:04:16.896856 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.896861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.896882 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:04:16.896973 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.896979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.897047 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.897148 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:16.897419 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.897912 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.898976 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.899595 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.899738 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.899789 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.899976 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.900042 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:04:16.900159 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.896598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.130:42943 (local address 127.2.208.131:59867)
0620 19:04:16.896747 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.896764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.896871 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.897158 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.897161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.897180 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:16.897300 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.897307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.898058 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.898061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.898844 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.898852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.899747 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.899750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.899845 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.899865 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.899962 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.900030 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":17}
I20260620 19:04:16.900667 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.896766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.208.131:50143 (local address 127.2.208.129:33765)
0620 19:04:16.896911 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.896917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.897276 (+ 359us) server_negotiation.cc:407] Connection header received
0620 19:04:16.897431 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.897434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.897486 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.897563 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:04:16.897834 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.898317 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.899705 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.900088 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.900239 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.900298 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.900485 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.900555 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:04:16.900856 22913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb06524ebb546b6aaa06d033a2db3cc" candidate_uuid: "703ba0cc10a547c398474c787d257d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72da029db4b9445a854367c00a2f1263" is_pre_election: true
I20260620 19:04:16.900982 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.896598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33765 (local address 127.2.208.131:50143)
0620 19:04:16.897191 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.897203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:16.897321 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:16.897583 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:16.897585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:16.897600 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:04:16.897721 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.897726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.898432 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.898436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.899589 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:16.899598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.900238 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.900242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:16.900353 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.900373 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:04:16.900803 (+ 430us) client_negotiation.cc:240] Negotiation successful
0620 19:04:16.900878 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":219,"thread_start_us":144,"threads_started":1}
I20260620 19:04:16.901206 22913 raft_consensus.cc:2454] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703ba0cc10a547c398474c787d257d78 in term 0.
I20260620 19:04:16.901576 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb06524ebb546b6aaa06d033a2db3cc" candidate_uuid: "703ba0cc10a547c398474c787d257d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" is_pre_election: true
I20260620 19:04:16.901758 22980 leader_election.cc:302] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703ba0cc10a547c398474c787d257d78, 72da029db4b9445a854367c00a2f1263; no voters:
I20260620 19:04:16.901850 22782 raft_consensus.cc:2454] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703ba0cc10a547c398474c787d257d78 in term 0.
I20260620 19:04:16.902029 23113 raft_consensus.cc:2781] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:04:16.902139 23113 raft_consensus.cc:482] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:04:16.902202 23113 raft_consensus.cc:3037] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:16.903232 23113 raft_consensus.cc:504] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.903613 23113 leader_election.cc:288] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [CANDIDATE]: Term 1 election: Requested vote from peers c5ceb0f0f8bf4de0aa665d58e4ebf345 (127.2.208.129:33765), 72da029db4b9445a854367c00a2f1263 (127.2.208.130:42943)
I20260620 19:04:16.903985 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb06524ebb546b6aaa06d033a2db3cc" candidate_uuid: "703ba0cc10a547c398474c787d257d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345"
I20260620 19:04:16.904004 22913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb06524ebb546b6aaa06d033a2db3cc" candidate_uuid: "703ba0cc10a547c398474c787d257d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72da029db4b9445a854367c00a2f1263"
I20260620 19:04:16.904114 22782 raft_consensus.cc:3037] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:16.904114 22913 raft_consensus.cc:3037] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:16.905407 22913 raft_consensus.cc:2454] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703ba0cc10a547c398474c787d257d78 in term 1.
I20260620 19:04:16.905407 22782 raft_consensus.cc:2454] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703ba0cc10a547c398474c787d257d78 in term 1.
I20260620 19:04:16.905743 22978 leader_election.cc:302] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703ba0cc10a547c398474c787d257d78, c5ceb0f0f8bf4de0aa665d58e4ebf345; no voters:
I20260620 19:04:16.905908 23113 raft_consensus.cc:2781] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:16.906140 23113 raft_consensus.cc:686] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 1 LEADER]: Becoming Leader. State: Replica: 703ba0cc10a547c398474c787d257d78, State: Running, Role: LEADER
I20260620 19:04:16.906514 23113 consensus_queue.cc:237] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } }
I20260620 19:04:16.909839 22640 catalog_manager.cc:5495] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703ba0cc10a547c398474c787d257d78 (127.2.208.131). New cstate: current_term: 1 leader_uuid: "703ba0cc10a547c398474c787d257d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703ba0cc10a547c398474c787d257d78" member_type: VOTER last_known_addr { host: "127.2.208.131" port: 39005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 } health_report { overall_health: UNKNOWN } } }
I20260620 19:04:16.936789 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.933226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.2.208.131:39005)
0620 19:04:16.933369 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:16.933373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:16.933386 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:04:16.933542 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:16.933544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:16.933594 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:16.933699 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:16.933929 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.934405 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.935279 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:16.935564 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:16.935675 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:16.935986 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:16.936378 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:16.936528 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:16.936578 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:04:16.936668 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:04:16.942633 22913 raft_consensus.cc:1261] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 1 FOLLOWER]: Refusing update from remote peer 703ba0cc10a547c398474c787d257d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:16.942632 22782 raft_consensus.cc:1261] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 1 FOLLOWER]: Refusing update from remote peer 703ba0cc10a547c398474c787d257d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:16.943477 23117 consensus_queue.cc:1035] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ceb0f0f8bf4de0aa665d58e4ebf345" member_type: VOTER last_known_addr { host: "127.2.208.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:16.943717 23113 consensus_queue.cc:1035] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72da029db4b9445a854367c00a2f1263" member_type: VOTER last_known_addr { host: "127.2.208.130" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:16.950461 23120 mvcc.cc:205] Tried to move back new op lower bound from 7298999324429639680 to 7298999324291784704. Current Snapshot: MvccSnapshot[applied={T|T < 7298999324429639680}]
I20260620 19:04:16.952376 23121 mvcc.cc:205] Tried to move back new op lower bound from 7298999324429639680 to 7298999324291784704. Current Snapshot: MvccSnapshot[applied={T|T < 7298999324429639680 or (T in {7298999324429639680})}]
I20260620 19:04:16.958591 22639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:16.958767 22639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:04:16.961112 22639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2848b10b76489b8a383d05651e8bbf: Sending DeleteTablet for 3 replicas of tablet 0bb06524ebb546b6aaa06d033a2db3cc
I20260620 19:04:16.961784 22762 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:16 UTC) from {username='slave'} at 127.0.0.1:51428
I20260620 19:04:16.961834 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:16 UTC) from {username='slave'} at 127.0.0.1:49834
I20260620 19:04:16.962280 23130 tablet_replica.cc:330] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345: stopping tablet replica
I20260620 19:04:16.962554 22893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb06524ebb546b6aaa06d033a2db3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:04:16 UTC) from {username='slave'} at 127.0.0.1:56036
I20260620 19:04:16.962631 23130 raft_consensus.cc:2229] T 0bb06524ebb546b6aaa06d033a2db3cc P c5ceb0f0f8bf4de0aa665d58e4ebf345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:16.962883 23131 tablet_replica.cc:330] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: stopping tablet replica
I20260620 19:04:16.963119 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22700
I20260620 19:04:16.963138 23131 raft_consensus.cc:2229] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:04:16.964856 23132 tablet_replica.cc:330] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: stopping tablet replica
I20260620 19:04:16.965126 23132 raft_consensus.cc:2229] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:04:16.965430 23132 raft_consensus.cc:2258] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:04:16.967105 23132 ts_tablet_manager.cc:1905] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:16.969769 23132 ts_tablet_manager.cc:1918] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:16.969862 23132 log.cc:1196] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-1/wal/wals/0bb06524ebb546b6aaa06d033a2db3cc
I20260620 19:04:16.970184 23132 ts_tablet_manager.cc:1939] T 0bb06524ebb546b6aaa06d033a2db3cc P 72da029db4b9445a854367c00a2f1263: Deleting consensus metadata
I20260620 19:04:16.970364 23131 raft_consensus.cc:2258] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:04:16.970412 22625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.208.129:33765 (error 108)
I20260620 19:04:16.971235 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22831
I20260620 19:04:16.971340 22627 catalog_manager.cc:4841] TS 72da029db4b9445a854367c00a2f1263 (127.2.208.130:42943): tablet 0bb06524ebb546b6aaa06d033a2db3cc (table test-table [id=5ed982175dd342ef8c206512527e2fc1]) successfully deleted
I20260620 19:04:16.972466 23131 ts_tablet_manager.cc:1905] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:04:16.975849 23131 ts_tablet_manager.cc:1918] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:04:16.975947 23131 log.cc:1196] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Deleting WAL directory at /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982008625501-2882-0/minicluster-data/ts-2/wal/wals/0bb06524ebb546b6aaa06d033a2db3cc
I20260620 19:04:16.976243 23131 ts_tablet_manager.cc:1939] T 0bb06524ebb546b6aaa06d033a2db3cc P 703ba0cc10a547c398474c787d257d78: Deleting consensus metadata
I20260620 19:04:16.977454 22624 catalog_manager.cc:4841] TS 703ba0cc10a547c398474c787d257d78 (127.2.208.131:39005): tablet 0bb06524ebb546b6aaa06d033a2db3cc (table test-table [id=5ed982175dd342ef8c206512527e2fc1]) successfully deleted
I20260620 19:04:16.977942 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:16.977542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.129:33765 (local address 127.0.0.1:51434)
0620 19:04:16.977763 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:16.977842 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 19:04:16.978116 22625 catalog_manager.cc:4584] TS c5ceb0f0f8bf4de0aa665d58e4ebf345 (127.2.208.129:33765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bb06524ebb546b6aaa06d033a2db3cc: Network error: Client connection negotiation failed: client connection to 127.2.208.129:33765: connect: Connection refused (error 111)
I20260620 19:04:16.979509 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22962
I20260620 19:04:16.985045 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 22609
2026-06-20T19:04:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1028 ms)
[----------] 6 tests from EncryptionPolicyTest (10335 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:17 dist-test-slave-tsm4 krb5kdc[23136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:17 dist-test-slave-tsm4 krb5kdc[23136](info): set up 2 sockets
Jun 20 19:04:17 dist-test-slave-tsm4 krb5kdc[23136](info): commencing operation
krb5kdc: starting...
W20260620 19:04:18.011464 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:18.124002 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:18.228600 23149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:18.228837 23149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:18.228876 23149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:18.232831 23149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:18.232880 23149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:18.232899 23149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:18.232924 23149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:18.232939 23149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:18.237911 23149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:18.238904 23149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:18.239933 23149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:18.247764 23154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:18.247756 23155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:18.247974 23149 server_base.cc:1034] running on GCE node
W20260620 19:04:18.247815 23157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:18.248600 23149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:18.249652 23149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:18.250840 23149 hybrid_clock.cc:648] HybridClock initialized: now 1781982258250817 us; error 41 us; skew 500 ppm
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:18.253409 23149 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:18.254223 23149 webserver.cc:460] Webserver started at http://127.0.0.1:35969/ using document root <none> and password file <none>
I20260620 19:04:18.254783 23149 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:18.254871 23149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:18.255096 23149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:18.257059 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "6685c0cefc9147a195fea8f7b17cf9b4"
format_stamp: "Formatted at 2026-06-20 19:04:18 on dist-test-slave-tsm4"
server_key: "87f413c5e75b7bb96e97920f14e8ab1d"
server_key_iv: "fbda182484c5f872ba5fa3a8ef52a995"
server_key_version: "encryptionkey@0"
I20260620 19:04:18.257618 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "6685c0cefc9147a195fea8f7b17cf9b4"
format_stamp: "Formatted at 2026-06-20 19:04:18 on dist-test-slave-tsm4"
server_key: "87f413c5e75b7bb96e97920f14e8ab1d"
server_key_iv: "fbda182484c5f872ba5fa3a8ef52a995"
server_key_version: "encryptionkey@0"
I20260620 19:04:18.261111 23149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:18.263458 23164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:18.264647 23149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:04:18.264779 23149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "6685c0cefc9147a195fea8f7b17cf9b4"
format_stamp: "Formatted at 2026-06-20 19:04:18 on dist-test-slave-tsm4"
server_key: "87f413c5e75b7bb96e97920f14e8ab1d"
server_key_iv: "fbda182484c5f872ba5fa3a8ef52a995"
server_key_version: "encryptionkey@0"
I20260620 19:04:18.264881 23149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:18.274606 23149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:18.277060 23149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:18.277246 23149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:18.284093 23149 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41759
I20260620 19:04:18.284121 23216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41759 every 8 connection(s)
I20260620 19:04:18.284994 23149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:18.287540 23217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:18.289247 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23149
I20260620 19:04:18.289369 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:18.289688 2882 external_mini_cluster.cc:1428] Setting key adde39efcd71519344bdb8253ec28137
I20260620 19:04:18.293596 23217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4: Bootstrap starting.
I20260620 19:04:18.295701 23217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:18.296378 23217 log.cc:824] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:04:18.298266 23217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4: No bootstrap required, opened a new log
I20260620 19:04:18.301278 23217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } }
I20260620 19:04:18.301535 23217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:18.301600 23217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6685c0cefc9147a195fea8f7b17cf9b4, State: Initialized, Role: FOLLOWER
I20260620 19:04:18.302057 23217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } }
I20260620 19:04:18.302205 23217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:18.302276 23217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:18.302408 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:18.303216 23217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } }
I20260620 19:04:18.303610 23217 leader_election.cc:302] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6685c0cefc9147a195fea8f7b17cf9b4; no voters:
I20260620 19:04:18.303999 23217 leader_election.cc:288] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:18.304113 23222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:18.304344 23222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [term 1 LEADER]: Becoming Leader. State: Replica: 6685c0cefc9147a195fea8f7b17cf9b4, State: Running, Role: LEADER
I20260620 19:04:18.304598 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:18.291017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39245 (local address 127.0.0.1:41759)
0620 19:04:18.291509 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:18.291523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:18.291555 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:04:18.292174 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:18.292190 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:18.292487 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:18.292793 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:18.293585 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:18.294557 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:18.295503 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:18.296019 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:18.296186 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:18.298863 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:18.298868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:18.298880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:18.298906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:18.301383 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:18.301972 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:18.301977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:18.301981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:18.302086 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:18.302473 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:18.302476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:18.302479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:18.302851 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:18.303079 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:18.303409 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 19:04:18.303617 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
I20260620 19:04:18.304651 23222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } }
I20260620 19:04:18.305248 23217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:18.305781 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6685c0cefc9147a195fea8f7b17cf9b4. Latest consensus state: current_term: 1 leader_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } } }
I20260620 19:04:18.305882 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:18.307488 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6685c0cefc9147a195fea8f7b17cf9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41759 } } }
I20260620 19:04:18.307873 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:18.307993 23228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:18.307659 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:18.310791 23228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:18.316383 23228 catalog_manager.cc:1347] Generated new cluster ID: 6c490008b20f4558a9a4854f4cd94803
I20260620 19:04:18.316439 23228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:18.324321 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:18.309695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35415 (local address 127.0.0.1:41759)
0620 19:04:18.309794 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:18.309799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:18.315648 (+ 5849us) server_negotiation.cc:407] Connection header received
0620 19:04:18.315802 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:18.315805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:18.315842 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:18.315921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:18.316742 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:18.317551 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:18.318660 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:18.319059 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:18.319169 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:18.320713 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:18.320717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:18.320720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:18.320744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:18.322450 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:18.322917 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:18.322921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:18.322923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:18.323006 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:18.323432 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:18.323435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:18.323437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:18.323619 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:18.323758 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:18.324092 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 19:04:18.324177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:04:18.342130 23228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:18.342975 23228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:18.347400 23228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6685c0cefc9147a195fea8f7b17cf9b4: Generated new TSK 0
I20260620 19:04:18.348026 23228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:18.354673 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23149
W20260620 19:04:18.360098 23134 connection.cc:535] client connection to 127.0.0.1:41759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41759): Connection reset by peer (error 104)
2026-06-20T19:04:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23248](info): set up 2 sockets
Jun 20 19:04:18 dist-test-slave-tsm4 krb5kdc[23248](info): commencing operation
krb5kdc: starting...
W20260620 19:04:19.391608 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:19 dist-test-slave-tsm4 krb5kdc[23248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:19.510303 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:19.616091 23261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:19.616333 23261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:19.616374 23261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:19.620251 23261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:19.620301 23261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:19.620319 23261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:19.620335 23261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:19.620350 23261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:19.625427 23261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:19.626314 23261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:19.627213 23261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:19.635253 23269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:19.635294 23267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:19.635490 23266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:19.635511 23261 server_base.cc:1034] running on GCE node
I20260620 19:04:19.636055 23261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:19.637063 23261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:19.638247 23261 hybrid_clock.cc:648] HybridClock initialized: now 1781982259638228 us; error 31 us; skew 500 ppm
Jun 20 19:04:19 dist-test-slave-tsm4 krb5kdc[23248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:19.640830 23261 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:19.641659 23261 webserver.cc:460] Webserver started at http://127.0.0.1:39997/ using document root <none> and password file <none>
I20260620 19:04:19.642189 23261 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:19.642274 23261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:19.642504 23261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:19.644523 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "3087c2cf77eb435f9c265de1eae75533"
format_stamp: "Formatted at 2026-06-20 19:04:19 on dist-test-slave-tsm4"
server_key: "1d6efa5ff37f64bc65a9b53b9c00c843"
server_key_iv: "8ba112f5d21b165c23e27d08408506d9"
server_key_version: "encryptionkey@0"
I20260620 19:04:19.645063 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "3087c2cf77eb435f9c265de1eae75533"
format_stamp: "Formatted at 2026-06-20 19:04:19 on dist-test-slave-tsm4"
server_key: "1d6efa5ff37f64bc65a9b53b9c00c843"
server_key_iv: "8ba112f5d21b165c23e27d08408506d9"
server_key_version: "encryptionkey@0"
I20260620 19:04:19.648566 23261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:19.650854 23276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:19.652081 23261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:19.652217 23261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "3087c2cf77eb435f9c265de1eae75533"
format_stamp: "Formatted at 2026-06-20 19:04:19 on dist-test-slave-tsm4"
server_key: "1d6efa5ff37f64bc65a9b53b9c00c843"
server_key_iv: "8ba112f5d21b165c23e27d08408506d9"
server_key_version: "encryptionkey@0"
I20260620 19:04:19.652323 23261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:19.669826 23261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:19.672497 23261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:19.672689 23261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:19.679574 23261 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45787
I20260620 19:04:19.679589 23328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45787 every 8 connection(s)
I20260620 19:04:19.680457 23261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:19.683065 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:19.685727 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23261
I20260620 19:04:19.685871 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:19.686180 2882 external_mini_cluster.cc:1428] Setting key 3744d075d9554e964f839f11b62ae269
I20260620 19:04:19.689025 23329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533: Bootstrap starting.
I20260620 19:04:19.691624 23329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:19.692382 23329 log.cc:824] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:19.694363 23329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533: No bootstrap required, opened a new log
I20260620 19:04:19.695838 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:19.687536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50361 (local address 127.0.0.1:45787)
0620 19:04:19.687937 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:19.687955 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:19.687999 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:04:19.688784 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:19.688805 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:19.689244 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:19.689577 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:19.689964 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:19.691450 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:19.692327 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:19.692934 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:19.693102 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:19.693910 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:19.693916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:19.693930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:19.694404 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:19.694469 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:19.694498 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:19.694623 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:19.694807 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:04:19.694931 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260620 19:04:19.697917 23329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260620 19:04:19.698141 23329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:19.698216 23329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3087c2cf77eb435f9c265de1eae75533, State: Initialized, Role: FOLLOWER
I20260620 19:04:19.698639 23329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260620 19:04:19.698805 23329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:19.698894 23329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:19.698963 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:19.698999 23329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:19.700075 23329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260620 19:04:19.700444 23329 leader_election.cc:302] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3087c2cf77eb435f9c265de1eae75533; no voters:
I20260620 19:04:19.700760 23329 leader_election.cc:288] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:19.700845 23338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:19.701110 23338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [term 1 LEADER]: Becoming Leader. State: Replica: 3087c2cf77eb435f9c265de1eae75533, State: Running, Role: LEADER
I20260620 19:04:19.701445 23338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } }
I20260620 19:04:19.702031 23329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:19.702711 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3087c2cf77eb435f9c265de1eae75533. Latest consensus state: current_term: 1 leader_uuid: "3087c2cf77eb435f9c265de1eae75533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } } }
I20260620 19:04:19.702821 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:19.703174 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3087c2cf77eb435f9c265de1eae75533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3087c2cf77eb435f9c265de1eae75533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45787 } } }
I20260620 19:04:19.703274 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:19.704519 23346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:19.707242 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:19.701253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60617 (local address 127.0.0.1:45787)
0620 19:04:19.701349 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:19.701354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:19.702129 (+ 775us) server_negotiation.cc:407] Connection header received
0620 19:04:19.702170 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:19.702173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:19.702215 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:19.702300 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:19.703609 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:19.704372 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:19.705395 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:19.705698 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:19.705805 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:19.706756 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:19.706760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:19.706763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:19.706775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:19.706801 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:19.706814 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:19.706904 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:19.707072 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:04:19.707110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:04:19.707872 23346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:19.712757 23346 catalog_manager.cc:1347] Generated new cluster ID: 5ac618634e274211aec1e1dca874d0de
I20260620 19:04:19.712821 23346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:19.725790 23346 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:19.726723 23346 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:19.730919 23346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3087c2cf77eb435f9c265de1eae75533: Generated new TSK 0
I20260620 19:04:19.731616 23346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:19.739202 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23261
W20260620 19:04:19.744652 23246 connection.cc:535] client connection to 127.0.0.1:45787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45787): Connection reset by peer (error 104)
2026-06-20T19:04:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:19 dist-test-slave-tsm4 krb5kdc[23360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:19 dist-test-slave-tsm4 krb5kdc[23360](info): set up 2 sockets
Jun 20 19:04:19 dist-test-slave-tsm4 krb5kdc[23360](info): commencing operation
krb5kdc: starting...
W20260620 19:04:20.775535 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:20 dist-test-slave-tsm4 krb5kdc[23360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:20.889861 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:20.995136 23373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:20.995453 23373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:20.995537 23373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:20.999442 23373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:20.999532 23373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:20.999567 23373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:20.999603 23373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:20.999637 23373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:21.004863 23373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:21.005896 23373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:21.006862 23373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:21.015007 23381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:21.015043 23378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:21.015002 23379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:21.015044 23373 server_base.cc:1034] running on GCE node
I20260620 19:04:21.015770 23373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:21.016824 23373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:21.017980 23373 hybrid_clock.cc:648] HybridClock initialized: now 1781982261017963 us; error 29 us; skew 500 ppm
Jun 20 19:04:21 dist-test-slave-tsm4 krb5kdc[23360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:21.020795 23373 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:21.021654 23373 webserver.cc:460] Webserver started at http://127.0.0.1:42039/ using document root <none> and password file <none>
I20260620 19:04:21.022207 23373 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:21.022265 23373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:21.022495 23373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:21.024547 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "463b67913c9a4071aa93e9d8b290dd17"
format_stamp: "Formatted at 2026-06-20 19:04:21 on dist-test-slave-tsm4"
server_key: "0501688392bb632e34e9026a202b8621"
server_key_iv: "fd62f7cb33d540906cd4b49a1a30e582"
server_key_version: "encryptionkey@0"
I20260620 19:04:21.025089 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "463b67913c9a4071aa93e9d8b290dd17"
format_stamp: "Formatted at 2026-06-20 19:04:21 on dist-test-slave-tsm4"
server_key: "0501688392bb632e34e9026a202b8621"
server_key_iv: "fd62f7cb33d540906cd4b49a1a30e582"
server_key_version: "encryptionkey@0"
I20260620 19:04:21.028609 23373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:21.030851 23388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:21.032052 23373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:21.032195 23373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "463b67913c9a4071aa93e9d8b290dd17"
format_stamp: "Formatted at 2026-06-20 19:04:21 on dist-test-slave-tsm4"
server_key: "0501688392bb632e34e9026a202b8621"
server_key_iv: "fd62f7cb33d540906cd4b49a1a30e582"
server_key_version: "encryptionkey@0"
I20260620 19:04:21.032301 23373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:21.036532 23373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:21.039156 23373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:21.039341 23373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:21.045926 23373 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43595
I20260620 19:04:21.045997 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43595 every 8 connection(s)
I20260620 19:04:21.046820 23373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:21.049444 23441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:21.054751 23441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: Bootstrap starting.
I20260620 19:04:21.055032 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23373
I20260620 19:04:21.055168 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:21.055501 2882 external_mini_cluster.cc:1428] Setting key 2f2b42a9b89149041ec328400a01ac0b
I20260620 19:04:21.057240 23441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:21.057965 23441 log.cc:824] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:21.060022 23441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: No bootstrap required, opened a new log
I20260620 19:04:21.060642 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:21.056741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38143 (local address 127.0.0.1:43595)
0620 19:04:21.057154 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:21.057170 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:21.057202 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:04:21.057923 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:21.057941 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:21.058280 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:21.058614 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:21.059431 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:21.059437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:21.059465 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:21.059475 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:21.059525 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:21.059544 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:21.059659 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:21.059758 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:04:21.059922 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260620 19:04:21.063315 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:21.063701 23441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260620 19:04:21.063903 23441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:21.063975 23441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463b67913c9a4071aa93e9d8b290dd17, State: Initialized, Role: FOLLOWER
I20260620 19:04:21.064393 23441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260620 19:04:21.064502 23441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:21.064630 23441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:21.064739 23441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:21.065776 23441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260620 19:04:21.066164 23441 leader_election.cc:302] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 463b67913c9a4071aa93e9d8b290dd17; no voters:
I20260620 19:04:21.066497 23441 leader_election.cc:288] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:21.066596 23451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:21.066850 23451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [term 1 LEADER]: Becoming Leader. State: Replica: 463b67913c9a4071aa93e9d8b290dd17, State: Running, Role: LEADER
I20260620 19:04:21.067334 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:21.065507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54453 (local address 127.0.0.1:43595)
0620 19:04:21.065619 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:21.065623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:21.065782 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:04:21.065953 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:21.065956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:21.066010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:21.066080 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:21.066879 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:21.066883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:21.066885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:21.066895 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:21.066918 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:21.066929 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:21.066995 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:21.067152 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 19:04:21.067208 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:21.067404 23451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260620 19:04:21.067895 23441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:21.068642 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 463b67913c9a4071aa93e9d8b290dd17. Latest consensus state: current_term: 1 leader_uuid: "463b67913c9a4071aa93e9d8b290dd17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } } }
I20260620 19:04:21.068759 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:21.069087 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "463b67913c9a4071aa93e9d8b290dd17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463b67913c9a4071aa93e9d8b290dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } } }
I20260620 19:04:21.069151 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17 [sys.catalog]: This master's current role is: LEADER
W20260620 19:04:21.072572 23466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:21.072685 23466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:21.072824 23460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:21.073639 23460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:21.078858 23460 catalog_manager.cc:1347] Generated new cluster ID: 645ca3cdddf845fca869eb825ae973c2
I20260620 19:04:21.078919 23460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:21.103632 23460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:21.103961 23460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:21.109238 23460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 463b67913c9a4071aa93e9d8b290dd17: Generated new TSK 0
I20260620 19:04:21.109992 23460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:21.122712 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23373
2026-06-20T19:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:21 dist-test-slave-tsm4 krb5kdc[23472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:21 dist-test-slave-tsm4 krb5kdc[23472](info): set up 2 sockets
Jun 20 19:04:21 dist-test-slave-tsm4 krb5kdc[23472](info): commencing operation
krb5kdc: starting...
W20260620 19:04:22.155608 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:22.273957 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:22.382223 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:22.382481 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:22.382577 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:22.386426 23485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:22.386504 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:22.386549 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:22.386610 23485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:22.386646 23485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:22.391724 23485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:22.392724 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:22.393659 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:22.401780 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:22.401773 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:22.401934 23485 server_base.cc:1034] running on GCE node
W20260620 19:04:22.401945 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:22.402593 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:22.403689 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:22.404884 23485 hybrid_clock.cc:648] HybridClock initialized: now 1781982262404849 us; error 37 us; skew 500 ppm
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:22.411572 23485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:22.412434 23485 webserver.cc:460] Webserver started at http://127.0.0.1:35443/ using document root <none> and password file <none>
I20260620 19:04:22.412971 23485 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:22.413055 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:22.413246 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:22.415307 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "135f004eb54c484485834db1f0bc13e5"
format_stamp: "Formatted at 2026-06-20 19:04:22 on dist-test-slave-tsm4"
server_key: "197d1255850bc85dad966a15d1976160"
server_key_iv: "1bf8e9f300c967a835e08122b2baf1cc"
server_key_version: "encryptionkey@0"
I20260620 19:04:22.415941 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "135f004eb54c484485834db1f0bc13e5"
format_stamp: "Formatted at 2026-06-20 19:04:22 on dist-test-slave-tsm4"
server_key: "197d1255850bc85dad966a15d1976160"
server_key_iv: "1bf8e9f300c967a835e08122b2baf1cc"
server_key_version: "encryptionkey@0"
I20260620 19:04:22.419590 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:22.422114 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:22.423296 23485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:22.423460 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "135f004eb54c484485834db1f0bc13e5"
format_stamp: "Formatted at 2026-06-20 19:04:22 on dist-test-slave-tsm4"
server_key: "197d1255850bc85dad966a15d1976160"
server_key_iv: "1bf8e9f300c967a835e08122b2baf1cc"
server_key_version: "encryptionkey@0"
I20260620 19:04:22.423570 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:22.439664 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:22.442289 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:22.442472 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:22.449184 23485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33201
I20260620 19:04:22.449196 23552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33201 every 8 connection(s)
I20260620 19:04:22.450130 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:22.452797 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:22.457993 23553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5: Bootstrap starting.
I20260620 19:04:22.459693 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23485
I20260620 19:04:22.459838 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:22.460114 2882 external_mini_cluster.cc:1428] Setting key 3357387faf21e27787bc403ffbbd4b4a
I20260620 19:04:22.460492 23553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:22.461416 23553 log.cc:824] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:22.463423 23553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5: No bootstrap required, opened a new log
I20260620 19:04:22.467444 23553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 19:04:22.467741 23553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:22.467814 23553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135f004eb54c484485834db1f0bc13e5, State: Initialized, Role: FOLLOWER
I20260620 19:04:22.468333 23553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 19:04:22.468531 23553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:22.468606 23553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:22.468743 23553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:04:22.469748 23553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 19:04:22.470134 23553 leader_election.cc:302] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135f004eb54c484485834db1f0bc13e5; no voters:
I20260620 19:04:22.470485 23553 leader_election.cc:288] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:22.470577 23558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:22.470834 23558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [term 1 LEADER]: Becoming Leader. State: Replica: 135f004eb54c484485834db1f0bc13e5, State: Running, Role: LEADER
I20260620 19:04:22.471172 23558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 19:04:22.471846 23553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:22.472782 23559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135f004eb54c484485834db1f0bc13e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260620 19:04:22.472893 23559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:22.473244 23560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135f004eb54c484485834db1f0bc13e5. Latest consensus state: current_term: 1 leader_uuid: "135f004eb54c484485834db1f0bc13e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f004eb54c484485834db1f0bc13e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260620 19:04:22.473417 23560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:22.474032 23567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:22.475656 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:22.461179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40781 (local address 127.0.0.1:33201)
0620 19:04:22.461590 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:22.461605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:22.461638 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:04:22.462414 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:22.462432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:22.462772 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:22.463076 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:22.463964 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:22.464935 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:22.466172 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:22.466626 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:22.466734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:22.469604 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:22.469613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:22.469633 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:22.469670 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:22.471575 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:22.472097 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:22.472105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:22.472114 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:22.472254 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:22.472628 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:22.472633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:22.472636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:22.473187 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:22.474096 (+ 909us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:22.474531 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 19:04:22.474793 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20260620 19:04:22.476644 23567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:22.478883 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:22.482263 23567 catalog_manager.cc:1347] Generated new cluster ID: c55bef58c836468a81e8c799c3f5ca00
I20260620 19:04:22.482322 23567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:22.489483 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:22.481126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40069 (local address 127.0.0.1:33201)
0620 19:04:22.481236 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:22.481240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:22.481460 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:04:22.481578 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:22.481581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:22.481623 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:22.481716 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:22.482426 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:22.483171 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:22.484319 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:22.484682 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:22.484747 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:22.486336 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:22.486340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:22.486342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:22.486364 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:22.487888 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:22.488313 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:22.488316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:22.488318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:22.488388 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:22.488648 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:22.488651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:22.488652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:22.488822 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:22.488907 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:22.489237 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 19:04:22.489335 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:04:22.499643 23567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:22.500705 23567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:22.506306 23567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135f004eb54c484485834db1f0bc13e5: Generated new TSK 0
I20260620 19:04:22.507019 23567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:22.514981 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23485
2026-06-20T19:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23584](info): set up 2 sockets
Jun 20 19:04:22 dist-test-slave-tsm4 krb5kdc[23584](info): commencing operation
krb5kdc: starting...
W20260620 19:04:23.547377 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:23 dist-test-slave-tsm4 krb5kdc[23584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:23.662281 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:23.768637 23597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:23.768904 23597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:23.768973 23597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:23.772930 23597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:23.773005 23597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:23.773041 23597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:23.773095 23597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:23.773123 23597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:23.778203 23597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:23.779175 23597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:23.780143 23597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:23.788041 23602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:23.788051 23603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:23.788084 23597 server_base.cc:1034] running on GCE node
W20260620 19:04:23.788051 23605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:23.788935 23597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:23.789893 23597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:23.791075 23597 hybrid_clock.cc:648] HybridClock initialized: now 1781982263791061 us; error 34 us; skew 500 ppm
Jun 20 19:04:23 dist-test-slave-tsm4 krb5kdc[23584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:23.793686 23597 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:23.794473 23597 webserver.cc:460] Webserver started at http://127.0.0.1:38185/ using document root <none> and password file <none>
I20260620 19:04:23.794981 23597 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:23.795066 23597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:23.795311 23597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:23.797257 23597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "55da8df00c3d4ff59ba07a4f429d1559"
format_stamp: "Formatted at 2026-06-20 19:04:23 on dist-test-slave-tsm4"
server_key: "9afb41792d9e6fdfbca3a0ab1d232e19"
server_key_iv: "2b90aeebf64521e7363027727948376a"
server_key_version: "encryptionkey@0"
I20260620 19:04:23.797791 23597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "55da8df00c3d4ff59ba07a4f429d1559"
format_stamp: "Formatted at 2026-06-20 19:04:23 on dist-test-slave-tsm4"
server_key: "9afb41792d9e6fdfbca3a0ab1d232e19"
server_key_iv: "2b90aeebf64521e7363027727948376a"
server_key_version: "encryptionkey@0"
I20260620 19:04:23.801267 23597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:23.803491 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:23.804641 23597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:04:23.804781 23597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "55da8df00c3d4ff59ba07a4f429d1559"
format_stamp: "Formatted at 2026-06-20 19:04:23 on dist-test-slave-tsm4"
server_key: "9afb41792d9e6fdfbca3a0ab1d232e19"
server_key_iv: "2b90aeebf64521e7363027727948376a"
server_key_version: "encryptionkey@0"
I20260620 19:04:23.804889 23597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:23.830476 23597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:23.833705 23597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:23.833907 23597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:23.840685 23597 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39199
I20260620 19:04:23.840711 23664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39199 every 8 connection(s)
I20260620 19:04:23.841575 23597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:23.844264 23665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:23.847476 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23597
I20260620 19:04:23.847612 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:23.847908 2882 external_mini_cluster.cc:1428] Setting key b0d16b5307b445f596898a8137090433
I20260620 19:04:23.850009 23665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559: Bootstrap starting.
I20260620 19:04:23.852175 23665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:23.852886 23665 log.cc:824] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:23.854646 23665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559: No bootstrap required, opened a new log
I20260620 19:04:23.856635 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:23.849234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34609 (local address 127.0.0.1:39199)
0620 19:04:23.849648 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:23.849662 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:23.849696 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:23.850346 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:23.850364 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:23.850689 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:23.851007 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:23.851462 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:23.852479 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:23.853393 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:23.853857 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:23.854024 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:23.854815 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:23.854822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:23.854844 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:23.855214 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:23.855262 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:23.855284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:23.855405 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:23.855565 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:04:23.855756 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260620 19:04:23.857695 23665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 19:04:23.857925 23665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:23.858002 23665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55da8df00c3d4ff59ba07a4f429d1559, State: Initialized, Role: FOLLOWER
I20260620 19:04:23.858471 23665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 19:04:23.858629 23665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:23.858701 23665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:23.858801 23665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:23.859256 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:23.859756 23665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 19:04:23.860064 23665 leader_election.cc:302] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55da8df00c3d4ff59ba07a4f429d1559; no voters:
I20260620 19:04:23.860369 23665 leader_election.cc:288] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:23.860431 23670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:23.860620 23670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [term 1 LEADER]: Becoming Leader. State: Replica: 55da8df00c3d4ff59ba07a4f429d1559, State: Running, Role: LEADER
I20260620 19:04:23.860885 23670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 19:04:23.861555 23665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:23.862022 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55da8df00c3d4ff59ba07a4f429d1559. Latest consensus state: current_term: 1 leader_uuid: "55da8df00c3d4ff59ba07a4f429d1559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260620 19:04:23.862197 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:23.862439 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55da8df00c3d4ff59ba07a4f429d1559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da8df00c3d4ff59ba07a4f429d1559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260620 19:04:23.862574 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:23.862862 23681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:23.865338 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:23.861228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33205 (local address 127.0.0.1:39199)
0620 19:04:23.861447 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:23.861453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:23.861467 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:23.861598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:23.861601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:23.861653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:23.861721 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:23.862013 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:23.862717 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:23.863827 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:23.864050 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:23.864155 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:23.864897 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:23.864901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:23.864903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:23.864915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:23.864939 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:23.864957 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:23.865031 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:23.865126 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:04:23.865171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 19:04:23.865906 23681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:23.871027 23681 catalog_manager.cc:1347] Generated new cluster ID: be8d2b654fa94c3f9563d2f98b56d983
I20260620 19:04:23.871095 23681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:23.900827 23681 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:23.902693 23681 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:23.908485 23681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55da8df00c3d4ff59ba07a4f429d1559: Generated new TSK 0
I20260620 19:04:23.909251 23681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:23.914100 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23597
2026-06-20T19:04:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:23 dist-test-slave-tsm4 krb5kdc[23696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:23 dist-test-slave-tsm4 krb5kdc[23696](info): set up 2 sockets
Jun 20 19:04:23 dist-test-slave-tsm4 krb5kdc[23696](info): commencing operation
krb5kdc: starting...
W20260620 19:04:24.931425 2882 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:25 dist-test-slave-tsm4 krb5kdc[23696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:25.050480 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:25.160468 23709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:25.160737 23709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:25.160805 23709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:25.164736 23709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:25.164815 23709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:25.164855 23709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:25.164891 23709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:25.164927 23709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:25.169986 23709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:25.170950 23709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:25.171919 23709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:25.179826 23715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:25.179858 23717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:25.179873 23714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:25.180083 23709 server_base.cc:1034] running on GCE node
I20260620 19:04:25.180697 23709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:25.181717 23709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:25.182896 23709 hybrid_clock.cc:648] HybridClock initialized: now 1781982265182874 us; error 46 us; skew 500 ppm
Jun 20 19:04:25 dist-test-slave-tsm4 krb5kdc[23696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:25.185570 23709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:25.186409 23709 webserver.cc:460] Webserver started at http://127.0.0.1:41219/ using document root <none> and password file <none>
I20260620 19:04:25.186934 23709 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:25.186996 23709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:25.187243 23709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:25.189275 23709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "9653c5f36da8484c87df3135cccf4e11"
format_stamp: "Formatted at 2026-06-20 19:04:25 on dist-test-slave-tsm4"
server_key: "f320cbfd649f3b221a482b9a2eb1d3ed"
server_key_iv: "c723b84f47dc2888d5e9e80484fd8a9b"
server_key_version: "encryptionkey@0"
I20260620 19:04:25.189836 23709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "9653c5f36da8484c87df3135cccf4e11"
format_stamp: "Formatted at 2026-06-20 19:04:25 on dist-test-slave-tsm4"
server_key: "f320cbfd649f3b221a482b9a2eb1d3ed"
server_key_iv: "c723b84f47dc2888d5e9e80484fd8a9b"
server_key_version: "encryptionkey@0"
I20260620 19:04:25.193289 23709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:25.195542 23724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:25.196651 23709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:25.196784 23709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "9653c5f36da8484c87df3135cccf4e11"
format_stamp: "Formatted at 2026-06-20 19:04:25 on dist-test-slave-tsm4"
server_key: "f320cbfd649f3b221a482b9a2eb1d3ed"
server_key_iv: "c723b84f47dc2888d5e9e80484fd8a9b"
server_key_version: "encryptionkey@0"
I20260620 19:04:25.196880 23709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:25.201187 23709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:25.204252 23709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:25.204435 23709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:25.210853 23709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41347
I20260620 19:04:25.210915 23776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41347 every 8 connection(s)
I20260620 19:04:25.211768 23709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:25.214216 23777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:25.215922 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23709
I20260620 19:04:25.216058 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:25.216352 2882 external_mini_cluster.cc:1428] Setting key d90ae1d74eb51108306201b0049bf9c7
I20260620 19:04:25.220244 23777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11: Bootstrap starting.
I20260620 19:04:25.221935 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:25.217735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52909 (local address 127.0.0.1:41347)
0620 19:04:25.218133 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:25.218151 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:25.218191 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:04:25.219176 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:25.219208 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:25.219630 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:25.219977 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:25.220607 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:25.220612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:25.220626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:25.220645 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:25.220700 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:25.220723 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:25.220862 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:25.220925 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:04:25.221074 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260620 19:04:25.222783 23777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:25.223635 23777 log.cc:824] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:25.224543 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:25.225533 23777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11: No bootstrap required, opened a new log
I20260620 19:04:25.228128 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:25.226481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40569 (local address 127.0.0.1:41347)
0620 19:04:25.226602 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:25.226606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:25.226798 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:04:25.226911 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:25.226913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:25.226940 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:25.227001 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:25.227752 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:25.227755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:25.227757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:25.227769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:25.227792 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:25.227802 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:25.227873 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:25.227975 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:04:25.228003 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:04:25.230044 23777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260620 19:04:25.230324 23777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:25.230386 23777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9653c5f36da8484c87df3135cccf4e11, State: Initialized, Role: FOLLOWER
I20260620 19:04:25.230921 23777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260620 19:04:25.231053 23777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:25.231132 23777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:25.231230 23777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:25.232183 23777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260620 19:04:25.232522 23777 leader_election.cc:302] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9653c5f36da8484c87df3135cccf4e11; no voters:
I20260620 19:04:25.232834 23777 leader_election.cc:288] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:25.232949 23787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:25.233152 23787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [term 1 LEADER]: Becoming Leader. State: Replica: 9653c5f36da8484c87df3135cccf4e11, State: Running, Role: LEADER
I20260620 19:04:25.233477 23787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } }
I20260620 19:04:25.234048 23777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:25.235255 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9653c5f36da8484c87df3135cccf4e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } } }
I20260620 19:04:25.235229 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9653c5f36da8484c87df3135cccf4e11. Latest consensus state: current_term: 1 leader_uuid: "9653c5f36da8484c87df3135cccf4e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9653c5f36da8484c87df3135cccf4e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41347 } } }
I20260620 19:04:25.235394 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:25.235394 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:25.235822 23796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:25.238801 23796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:25.243961 23796 catalog_manager.cc:1347] Generated new cluster ID: c7eb229b036f4f43a988df6ae4dbf043
I20260620 19:04:25.244019 23796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:25.261415 23796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:25.261746 23796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:25.269479 23796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9653c5f36da8484c87df3135cccf4e11: Generated new TSK 0
I20260620 19:04:25.270079 23796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:25.277029 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23709
2026-06-20T19:04:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:25 dist-test-slave-tsm4 krb5kdc[23808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:25 dist-test-slave-tsm4 krb5kdc[23808](info): set up 2 sockets
Jun 20 19:04:25 dist-test-slave-tsm4 krb5kdc[23808](info): commencing operation
krb5kdc: starting...
W20260620 19:04:26.295610 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:26.415096 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:26.522495 23821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:26.522789 23821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:26.522878 23821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:26.526902 23821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:26.526980 23821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:26.527030 23821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:26.527086 23821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:26.527123 23821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:26.532461 23821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:26.533470 23821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:26.534452 23821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:26.542768 23829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:26.542768 23826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:26.542789 23827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:26.543010 23821 server_base.cc:1034] running on GCE node
I20260620 19:04:26.543738 23821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:26.544768 23821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:26.545926 23821 hybrid_clock.cc:648] HybridClock initialized: now 1781982266545914 us; error 24 us; skew 500 ppm
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:26.548553 23821 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:26.549382 23821 webserver.cc:460] Webserver started at http://127.0.0.1:40331/ using document root <none> and password file <none>
I20260620 19:04:26.549906 23821 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:26.549966 23821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:26.550196 23821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:26.552214 23821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "61cbfafed0a3471ea8b1de28541a7f6f"
format_stamp: "Formatted at 2026-06-20 19:04:26 on dist-test-slave-tsm4"
server_key: "7f9275bf93ed2c1518f4f2b6e94da3e3"
server_key_iv: "83f7ee4cc31133b2f61cc56f96ebe418"
server_key_version: "encryptionkey@0"
I20260620 19:04:26.552748 23821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "61cbfafed0a3471ea8b1de28541a7f6f"
format_stamp: "Formatted at 2026-06-20 19:04:26 on dist-test-slave-tsm4"
server_key: "7f9275bf93ed2c1518f4f2b6e94da3e3"
server_key_iv: "83f7ee4cc31133b2f61cc56f96ebe418"
server_key_version: "encryptionkey@0"
I20260620 19:04:26.556294 23821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:26.558578 23836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:26.559787 23821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:26.559922 23821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "61cbfafed0a3471ea8b1de28541a7f6f"
format_stamp: "Formatted at 2026-06-20 19:04:26 on dist-test-slave-tsm4"
server_key: "7f9275bf93ed2c1518f4f2b6e94da3e3"
server_key_iv: "83f7ee4cc31133b2f61cc56f96ebe418"
server_key_version: "encryptionkey@0"
I20260620 19:04:26.560029 23821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:26.584293 23821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:26.587013 23821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:26.587224 23821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:26.594058 23821 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33123
I20260620 19:04:26.594089 23888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33123 every 8 connection(s)
I20260620 19:04:26.594944 23821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:26.597568 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:26.600725 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23821
I20260620 19:04:26.600842 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:26.601133 2882 external_mini_cluster.cc:1428] Setting key 55b85f95b9c7063f32ded89cc36789c9
I20260620 19:04:26.603199 23889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: Bootstrap starting.
I20260620 19:04:26.605314 23889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:26.606012 23889 log.cc:824] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:26.607826 23889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: No bootstrap required, opened a new log
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:04:26.611850 23889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } }
I20260620 19:04:26.612049 23889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:26.612107 23889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61cbfafed0a3471ea8b1de28541a7f6f, State: Initialized, Role: FOLLOWER
I20260620 19:04:26.612578 23889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } }
I20260620 19:04:26.612725 23889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:26.612821 23889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:26.612941 23889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:26.613873 23889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } }
I20260620 19:04:26.614248 23889 leader_election.cc:302] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61cbfafed0a3471ea8b1de28541a7f6f; no voters:
I20260620 19:04:26.614599 23889 leader_election.cc:288] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:26.614672 23894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:26.614851 23894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [term 1 LEADER]: Becoming Leader. State: Replica: 61cbfafed0a3471ea8b1de28541a7f6f, State: Running, Role: LEADER
I20260620 19:04:26.615188 23894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } }
I20260620 19:04:26.615407 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:26.602452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:42109 (local address 127.0.0.1:33123)
0620 19:04:26.602856 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:26.602870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:26.602904 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:26.603605 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:26.603630 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:26.603947 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:26.604266 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:26.605018 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:26.606158 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:26.607093 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:26.607626 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:26.607787 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:26.610458 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:26.610463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:26.610473 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:26.610498 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:26.612175 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:26.612673 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:26.612678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:26.612684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:26.612879 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:26.613261 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:26.613264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:26.613265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:26.613608 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:26.613767 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:26.614174 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 19:04:26.614419 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260620 19:04:26.615840 23889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:26.618227 23895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } } }
I20260620 19:04:26.618342 23895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:26.618737 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:26.618636 23896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61cbfafed0a3471ea8b1de28541a7f6f. Latest consensus state: current_term: 1 leader_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61cbfafed0a3471ea8b1de28541a7f6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33123 } } }
I20260620 19:04:26.618878 23896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f [sys.catalog]: This master's current role is: LEADER
W20260620 19:04:26.621068 23909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:26.621158 23909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:26.621243 23910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:26.621933 23910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:26.627382 23910 catalog_manager.cc:1347] Generated new cluster ID: 7d8a03b2cf54485d9bf9b292e884c8ca
I20260620 19:04:26.627441 23910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:26.631475 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:26.620915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:44421 (local address 127.0.0.1:33123)
0620 19:04:26.621007 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:26.621011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:26.623624 (+ 2613us) server_negotiation.cc:407] Connection header received
0620 19:04:26.623758 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:26.623763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:26.623805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:26.623887 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:26.624520 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:26.625040 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:26.626078 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:26.626451 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:26.626558 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:26.628117 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:26.628120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:26.628122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:26.628142 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:26.629588 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:26.630055 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:26.630061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:26.630067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:26.630136 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:26.630571 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:26.630574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:26.630576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:26.630747 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:26.630837 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:26.631176 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 19:04:26.631265 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:04:26.645510 23910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:26.646447 23910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:26.650960 23910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61cbfafed0a3471ea8b1de28541a7f6f: Generated new TSK 0
I20260620 19:04:26.651674 23910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:26.664598 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23821
W20260620 19:04:26.669548 23806 connection.cc:535] client connection to 127.0.0.1:33123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33123): Connection reset by peer (error 104)
2026-06-20T19:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23920](info): set up 2 sockets
Jun 20 19:04:26 dist-test-slave-tsm4 krb5kdc[23920](info): commencing operation
krb5kdc: starting...
W20260620 19:04:27.695595 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:27 dist-test-slave-tsm4 krb5kdc[23920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:27.810559 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:27.917302 23933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:27.917611 23933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:27.917686 23933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:27.921778 23933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:27.921869 23933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:27.921907 23933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:27.921965 23933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:27.922004 23933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:27.927233 23933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:27.928320 23933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:27.929354 23933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:27.937881 23941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:27.937925 23933 server_base.cc:1034] running on GCE node
W20260620 19:04:27.937862 23939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:27.937871 23938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:27.938684 23933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:27.939836 23933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:27.940994 23933 hybrid_clock.cc:648] HybridClock initialized: now 1781982267940977 us; error 32 us; skew 500 ppm
Jun 20 19:04:27 dist-test-slave-tsm4 krb5kdc[23920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:27.947615 23933 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:27.948559 23933 webserver.cc:460] Webserver started at http://127.0.0.1:41849/ using document root <none> and password file <none>
I20260620 19:04:27.949178 23933 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:27.949268 23933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:27.949501 23933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:27.951584 23933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "ea6be75dfaf748e1866169f6c630b102"
format_stamp: "Formatted at 2026-06-20 19:04:27 on dist-test-slave-tsm4"
server_key: "545416e7327b07c42a87fa96f4347fc9"
server_key_iv: "912974eb06e89eeb258ee66a8bb4e118"
server_key_version: "encryptionkey@0"
I20260620 19:04:27.952144 23933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "ea6be75dfaf748e1866169f6c630b102"
format_stamp: "Formatted at 2026-06-20 19:04:27 on dist-test-slave-tsm4"
server_key: "545416e7327b07c42a87fa96f4347fc9"
server_key_iv: "912974eb06e89eeb258ee66a8bb4e118"
server_key_version: "encryptionkey@0"
I20260620 19:04:27.955708 23933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:27.958078 23948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:27.959498 23933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:27.959628 23933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "ea6be75dfaf748e1866169f6c630b102"
format_stamp: "Formatted at 2026-06-20 19:04:27 on dist-test-slave-tsm4"
server_key: "545416e7327b07c42a87fa96f4347fc9"
server_key_iv: "912974eb06e89eeb258ee66a8bb4e118"
server_key_version: "encryptionkey@0"
I20260620 19:04:27.959744 23933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:27.975329 23933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:27.978196 23933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:27.978397 23933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:27.985549 23933 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39951
I20260620 19:04:27.985572 24000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39951 every 8 connection(s)
I20260620 19:04:27.986498 23933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:27.989241 24001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:27.994545 24001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102: Bootstrap starting.
I20260620 19:04:27.996248 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 23933
I20260620 19:04:27.996402 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:27.996677 2882 external_mini_cluster.cc:1428] Setting key 7e7e3ccd18512dee00add0bcde1e55e3
I20260620 19:04:27.997189 24001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:27.998085 24001 log.cc:824] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:28.000021 24001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102: No bootstrap required, opened a new log
I20260620 19:04:28.003412 24001 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260620 19:04:28.003660 24001 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:28.003841 24001 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6be75dfaf748e1866169f6c630b102, State: Initialized, Role: FOLLOWER
I20260620 19:04:28.004297 24001 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260620 19:04:28.004452 24001 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:28.004523 24001 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:28.004630 24001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:28.005507 24001 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260620 19:04:28.005594 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:27.997734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:47919 (local address 127.0.0.1:39951)
0620 19:04:27.998136 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:27.998150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:27.998183 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:04:27.999027 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:27.999052 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:27.999461 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:27.999789 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:28.000252 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:28.001195 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:28.002110 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:28.002582 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:28.002752 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:28.003680 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:28.003687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:28.003703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:28.004222 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:28.004282 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:28.004306 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:28.004443 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:28.004758 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 19:04:28.004886 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260620 19:04:28.005892 24001 leader_election.cc:302] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea6be75dfaf748e1866169f6c630b102; no voters:
I20260620 19:04:28.006210 24001 leader_election.cc:288] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:28.006301 24006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:28.006517 24006 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [term 1 LEADER]: Becoming Leader. State: Replica: ea6be75dfaf748e1866169f6c630b102, State: Running, Role: LEADER
I20260620 19:04:28.006837 24006 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } }
I20260620 19:04:28.007457 24001 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:28.008170 24007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea6be75dfaf748e1866169f6c630b102. Latest consensus state: current_term: 1 leader_uuid: "ea6be75dfaf748e1866169f6c630b102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } } }
I20260620 19:04:28.008352 24007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:28.008548 24008 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea6be75dfaf748e1866169f6c630b102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6be75dfaf748e1866169f6c630b102" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39951 } } }
I20260620 19:04:28.008656 24008 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:28.008945 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:28.009085 24015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:28.011986 24015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:28.017612 24015 catalog_manager.cc:1347] Generated new cluster ID: ae4c41ac4ac1489f93c89ffad4062235
I20260620 19:04:28.017666 24015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:28.020361 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:28.015915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:41707 (local address 127.0.0.1:39951)
0620 19:04:28.016054 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:28.016075 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:28.016462 (+ 387us) server_negotiation.cc:407] Connection header received
0620 19:04:28.016581 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:28.016584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:28.016634 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:28.016719 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:28.017028 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:28.017764 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:28.018625 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:28.018874 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:28.018977 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:28.019882 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:28.019885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:28.019888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:28.019901 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:28.019934 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:28.019944 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:28.020040 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:28.020166 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:04:28.020197 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:04:28.033923 24015 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:28.034814 24015 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:28.038391 24015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea6be75dfaf748e1866169f6c630b102: Generated new TSK 0
I20260620 19:04:28.039119 24015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:28.046425 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 23933
W20260620 19:04:28.051752 23918 connection.cc:535] client connection to 127.0.0.1:39951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39951): Connection reset by peer (error 104)
2026-06-20T19:04:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:28 dist-test-slave-tsm4 krb5kdc[24032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:28 dist-test-slave-tsm4 krb5kdc[24032](info): set up 2 sockets
Jun 20 19:04:28 dist-test-slave-tsm4 krb5kdc[24032](info): commencing operation
krb5kdc: starting...
W20260620 19:04:29.067845 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:29 dist-test-slave-tsm4 krb5kdc[24032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:29.184332 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:29.293030 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:29.293308 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:29.293377 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:29.297268 24045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:29.297359 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:29.297396 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:29.297451 24045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:29.297488 24045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:29.302520 24045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:29.303545 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:29.304538 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:29.313023 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:29.313023 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:29.313025 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:29.313298 24045 server_base.cc:1034] running on GCE node
I20260620 19:04:29.313900 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:29.315596 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:29.316757 24045 hybrid_clock.cc:648] HybridClock initialized: now 1781982269316741 us; error 38 us; skew 500 ppm
Jun 20 19:04:29 dist-test-slave-tsm4 krb5kdc[24032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:29.321758 24045 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:29.322979 24045 webserver.cc:460] Webserver started at http://127.0.0.1:38731/ using document root <none> and password file <none>
I20260620 19:04:29.323720 24045 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:29.323783 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:29.324007 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:29.326079 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "bc5238f0227243f093700164238ebd13"
format_stamp: "Formatted at 2026-06-20 19:04:29 on dist-test-slave-tsm4"
server_key: "6dd249ae766d1b6d4d90e27b3fe4c9ff"
server_key_iv: "fffb4d5a6c664e1c56ae699bbe0fc8d4"
server_key_version: "encryptionkey@0"
I20260620 19:04:29.326644 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "bc5238f0227243f093700164238ebd13"
format_stamp: "Formatted at 2026-06-20 19:04:29 on dist-test-slave-tsm4"
server_key: "6dd249ae766d1b6d4d90e27b3fe4c9ff"
server_key_iv: "fffb4d5a6c664e1c56ae699bbe0fc8d4"
server_key_version: "encryptionkey@0"
I20260620 19:04:29.330273 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:04:29.332677 24060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:29.333884 24045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:29.334024 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "bc5238f0227243f093700164238ebd13"
format_stamp: "Formatted at 2026-06-20 19:04:29 on dist-test-slave-tsm4"
server_key: "6dd249ae766d1b6d4d90e27b3fe4c9ff"
server_key_iv: "fffb4d5a6c664e1c56ae699bbe0fc8d4"
server_key_version: "encryptionkey@0"
I20260620 19:04:29.334163 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:29.338828 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:29.341548 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:29.341741 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:29.348322 24045 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38289
I20260620 19:04:29.348338 24112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38289 every 8 connection(s)
I20260620 19:04:29.349200 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:29.349956 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24045
I20260620 19:04:29.350106 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:29.350411 2882 external_mini_cluster.cc:1428] Setting key 47f863845c47314767bac85115cee3d5
I20260620 19:04:29.352917 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:29.355242 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:29.351654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:46021 (local address 127.0.0.1:38289)
0620 19:04:29.352015 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:29.352028 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:29.352059 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:04:29.352703 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:29.352718 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:29.353006 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:29.353297 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:29.354066 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:29.354070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:29.354080 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:29.354090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:29.354145 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:29.354164 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:29.354266 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:29.354398 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:04:29.354563 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260620 19:04:29.357995 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:29.359170 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13: Bootstrap starting.
I20260620 19:04:29.361501 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:29.360029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:60291 (local address 127.0.0.1:38289)
0620 19:04:29.360148 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:29.360152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:29.360301 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:04:29.360376 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:29.360381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:29.360418 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:29.360498 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:29.361150 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:29.361154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:29.361156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:29.361167 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:29.361192 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:29.361204 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:29.361293 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:29.361346 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:04:29.361380 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:29.361673 24113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:29.362457 24113 log.cc:824] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:29.364168 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13: No bootstrap required, opened a new log
I20260620 19:04:29.367583 24113 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260620 19:04:29.367825 24113 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:29.367901 24113 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5238f0227243f093700164238ebd13, State: Initialized, Role: FOLLOWER
I20260620 19:04:29.368323 24113 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260620 19:04:29.368440 24113 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:29.368479 24113 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:29.368546 24113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:29.369329 24113 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260620 19:04:29.369601 24113 leader_election.cc:302] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5238f0227243f093700164238ebd13; no voters:
I20260620 19:04:29.369834 24113 leader_election.cc:288] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:29.369947 24123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:29.370219 24123 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [term 1 LEADER]: Becoming Leader. State: Replica: bc5238f0227243f093700164238ebd13, State: Running, Role: LEADER
I20260620 19:04:29.370520 24123 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260620 19:04:29.371002 24113 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:29.372108 24125 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5238f0227243f093700164238ebd13. Latest consensus state: current_term: 1 leader_uuid: "bc5238f0227243f093700164238ebd13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } } }
I20260620 19:04:29.372238 24125 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:29.372596 24124 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc5238f0227243f093700164238ebd13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5238f0227243f093700164238ebd13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } } }
I20260620 19:04:29.372689 24124 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:29.372655 24132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:29.375710 24132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:29.380864 24132 catalog_manager.cc:1347] Generated new cluster ID: 64275325980748b784769b3a0c46d897
I20260620 19:04:29.380926 24132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:29.409555 24132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:29.409890 24132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:29.421996 24132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc5238f0227243f093700164238ebd13: Generated new TSK 0
I20260620 19:04:29.422698 24132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:29.428939 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24045
2026-06-20T19:04:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:29 dist-test-slave-tsm4 krb5kdc[24144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:29 dist-test-slave-tsm4 krb5kdc[24144](info): set up 2 sockets
Jun 20 19:04:29 dist-test-slave-tsm4 krb5kdc[24144](info): commencing operation
krb5kdc: starting...
W20260620 19:04:30.455542 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:30.570266 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:30.676172 24157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:30.676409 24157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:30.676450 24157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:30.680322 24157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:30.680370 24157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:30.680389 24157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:30.680406 24157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:30.680421 24157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:30.685410 24157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:30.686286 24157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:30.687191 24157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:30.695188 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:30.695185 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:30.695182 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:30.695254 24157 server_base.cc:1034] running on GCE node
I20260620 19:04:30.696062 24157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:30.697103 24157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:30.698280 24157 hybrid_clock.cc:648] HybridClock initialized: now 1781982270698261 us; error 41 us; skew 500 ppm
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:30.700941 24157 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:30.701766 24157 webserver.cc:460] Webserver started at http://127.0.0.1:46441/ using document root <none> and password file <none>
I20260620 19:04:30.702279 24157 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:30.702373 24157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:30.702586 24157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:30.704535 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "5ae8bedae03e42af9dd75b1788bb98fc"
format_stamp: "Formatted at 2026-06-20 19:04:30 on dist-test-slave-tsm4"
server_key: "162d17fc3da423ee3bf76cab41c244e7"
server_key_iv: "b3b31703a3031006916935d617f0d38b"
server_key_version: "encryptionkey@0"
I20260620 19:04:30.705066 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "5ae8bedae03e42af9dd75b1788bb98fc"
format_stamp: "Formatted at 2026-06-20 19:04:30 on dist-test-slave-tsm4"
server_key: "162d17fc3da423ee3bf76cab41c244e7"
server_key_iv: "b3b31703a3031006916935d617f0d38b"
server_key_version: "encryptionkey@0"
I20260620 19:04:30.708436 24157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:30.710680 24172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:30.711874 24157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:30.712001 24157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "5ae8bedae03e42af9dd75b1788bb98fc"
format_stamp: "Formatted at 2026-06-20 19:04:30 on dist-test-slave-tsm4"
server_key: "162d17fc3da423ee3bf76cab41c244e7"
server_key_iv: "b3b31703a3031006916935d617f0d38b"
server_key_version: "encryptionkey@0"
I20260620 19:04:30.712113 24157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:30.730698 24157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:30.733855 24157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:30.734051 24157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:30.740988 24157 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39113
I20260620 19:04:30.741008 24224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39113 every 8 connection(s)
I20260620 19:04:30.741890 24157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:30.744617 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:30.745786 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24157
I20260620 19:04:30.745934 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:30.746219 2882 external_mini_cluster.cc:1428] Setting key 3c073dd6178e09c411dd46816be86ecd
I20260620 19:04:30.750746 24225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc: Bootstrap starting.
I20260620 19:04:30.753001 24225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:30.753702 24225 log.cc:824] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:04:30.755935 24225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc: No bootstrap required, opened a new log
I20260620 19:04:30.759840 24225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260620 19:04:30.760052 24225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:30.760111 24225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae8bedae03e42af9dd75b1788bb98fc, State: Initialized, Role: FOLLOWER
I20260620 19:04:30.760380 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:30.747655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:60211 (local address 127.0.0.1:39113)
0620 19:04:30.748032 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:30.748056 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:30.748089 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:04:30.748786 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:30.748810 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:30.749154 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:30.749486 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:30.750255 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:30.751244 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:30.752115 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:30.752588 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:30.752812 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:30.755683 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:30.755688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:30.755701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:30.755725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:30.757372 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:30.758000 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:30.758005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:30.758009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:30.758109 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:30.758432 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:30.758434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:30.758435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:30.758792 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:30.758963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:30.759286 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 19:04:30.759550 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260620 19:04:30.760589 24225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260620 19:04:30.760753 24225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:30.760838 24225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:30.760950 24225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:30.762006 24225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260620 19:04:30.762378 24225 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae8bedae03e42af9dd75b1788bb98fc; no voters:
I20260620 19:04:30.762738 24225 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:30.762848 24230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:30.763075 24230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [term 1 LEADER]: Becoming Leader. State: Replica: 5ae8bedae03e42af9dd75b1788bb98fc, State: Running, Role: LEADER
I20260620 19:04:30.763430 24230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260620 19:04:30.763556 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:30.763998 24225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:30.765246 24231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260620 19:04:30.765362 24231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:30.765667 24232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae8bedae03e42af9dd75b1788bb98fc. Latest consensus state: current_term: 1 leader_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae8bedae03e42af9dd75b1788bb98fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260620 19:04:30.765769 24243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:30.765836 24232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:30.769132 24243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:30.775195 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:30.766787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:54049 (local address 127.0.0.1:39113)
0620 19:04:30.766895 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:30.766899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:30.767218 (+ 319us) server_negotiation.cc:407] Connection header received
0620 19:04:30.767340 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:30.767343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:30.767423 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:30.767510 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:30.768110 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:30.768604 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:30.769553 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:30.769971 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:30.770063 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:30.771669 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:30.771673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:30.771675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:30.771698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:30.773246 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:30.773892 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:30.773896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:30.773898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:30.773979 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:30.774340 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:30.774343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:30.774345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:30.774537 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:30.774677 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:30.774953 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:04:30.775044 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:04:30.775228 24243 catalog_manager.cc:1347] Generated new cluster ID: f99b6b1fbc5c41df97256cb7ac860f8c
I20260620 19:04:30.775290 24243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:30.784570 24243 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:30.785532 24243 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:30.791185 24243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae8bedae03e42af9dd75b1788bb98fc: Generated new TSK 0
I20260620 19:04:30.791879 24243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:30.797381 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24157
W20260620 19:04:30.802652 24142 connection.cc:535] client connection to 127.0.0.1:39113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39113): Connection reset by peer (error 104)
2026-06-20T19:04:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24256](info): set up 2 sockets
Jun 20 19:04:30 dist-test-slave-tsm4 krb5kdc[24256](info): commencing operation
krb5kdc: starting...
W20260620 19:04:31.831525 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:31 dist-test-slave-tsm4 krb5kdc[24256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:31.952220 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:32.059144 24269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:32.059458 24269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:32.059536 24269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:32.063385 24269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:32.063467 24269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:32.063503 24269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:32.063580 24269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:32.063616 24269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:32.068559 24269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:32.069595 24269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:32.070585 24269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:32.078863 24277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:32.078891 24274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:32.078886 24275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:32.079034 24269 server_base.cc:1034] running on GCE node
I20260620 19:04:32.079607 24269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:32.080605 24269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:32.081801 24269 hybrid_clock.cc:648] HybridClock initialized: now 1781982272081788 us; error 24 us; skew 500 ppm
Jun 20 19:04:32 dist-test-slave-tsm4 krb5kdc[24256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:32.084443 24269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:32.085227 24269 webserver.cc:460] Webserver started at http://127.0.0.1:36057/ using document root <none> and password file <none>
I20260620 19:04:32.085760 24269 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:32.085845 24269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:32.086076 24269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:32.088146 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "d0f88d3d2be748bfa26a3d6655ffc433"
format_stamp: "Formatted at 2026-06-20 19:04:32 on dist-test-slave-tsm4"
server_key: "3cb304ad0f4a595d5245ba274224c09d"
server_key_iv: "cade77fd57e4079e769797e475131359"
server_key_version: "encryptionkey@0"
I20260620 19:04:32.088675 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "d0f88d3d2be748bfa26a3d6655ffc433"
format_stamp: "Formatted at 2026-06-20 19:04:32 on dist-test-slave-tsm4"
server_key: "3cb304ad0f4a595d5245ba274224c09d"
server_key_iv: "cade77fd57e4079e769797e475131359"
server_key_version: "encryptionkey@0"
I20260620 19:04:32.092098 24269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:32.094401 24284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:32.095574 24269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:32.095695 24269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "d0f88d3d2be748bfa26a3d6655ffc433"
format_stamp: "Formatted at 2026-06-20 19:04:32 on dist-test-slave-tsm4"
server_key: "3cb304ad0f4a595d5245ba274224c09d"
server_key_iv: "cade77fd57e4079e769797e475131359"
server_key_version: "encryptionkey@0"
I20260620 19:04:32.095793 24269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:32.120589 24269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:32.123400 24269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:32.123601 24269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:32.130942 24269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32925
I20260620 19:04:32.130962 24336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32925 every 8 connection(s)
I20260620 19:04:32.131964 24269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:32.134605 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:32.138314 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24269
I20260620 19:04:32.138454 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:32.138724 2882 external_mini_cluster.cc:1428] Setting key 16992e8725607377786f900d680eeab7
I20260620 19:04:32.140404 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433: Bootstrap starting.
I20260620 19:04:32.143029 24337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:32.143890 24337 log.cc:824] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:32.145884 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433: No bootstrap required, opened a new log
I20260620 19:04:32.148648 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:32.140160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:59471 (local address 127.0.0.1:32925)
0620 19:04:32.140554 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:32.140572 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:32.140614 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:04:32.141401 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:32.141422 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:32.141880 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:32.142234 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:32.142642 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:32.144147 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:32.145014 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:32.145670 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:32.145815 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:32.146687 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:32.146693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:32.146708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:32.147200 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:32.147262 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:32.147304 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:32.147468 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:32.147585 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:04:32.147721 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260620 19:04:32.150207 24337 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } }
I20260620 19:04:32.150481 24337 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:32.150533 24337 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f88d3d2be748bfa26a3d6655ffc433, State: Initialized, Role: FOLLOWER
I20260620 19:04:32.151026 24337 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } }
I20260620 19:04:32.151196 24337 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:32.151260 24337 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:32.151353 24337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:32.151575 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:32.152261 24337 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } }
I20260620 19:04:32.152587 24337 leader_election.cc:302] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f88d3d2be748bfa26a3d6655ffc433; no voters:
I20260620 19:04:32.152913 24337 leader_election.cc:288] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:32.152953 24343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:32.153264 24343 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [term 1 LEADER]: Becoming Leader. State: Replica: d0f88d3d2be748bfa26a3d6655ffc433, State: Running, Role: LEADER
I20260620 19:04:32.153612 24343 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } }
I20260620 19:04:32.154178 24337 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:32.154770 24343 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f88d3d2be748bfa26a3d6655ffc433. Latest consensus state: current_term: 1 leader_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } } }
I20260620 19:04:32.154954 24343 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:32.155203 24347 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f88d3d2be748bfa26a3d6655ffc433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32925 } } }
I20260620 19:04:32.155351 24347 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:32.155483 24356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:32.158473 24356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:32.160989 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:32.153778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:38811 (local address 127.0.0.1:32925)
0620 19:04:32.153873 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:32.153877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:32.156490 (+ 2613us) server_negotiation.cc:407] Connection header received
0620 19:04:32.156705 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:32.156711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:32.156762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:32.156841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:32.157303 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:32.158094 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:32.159231 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:32.159561 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:32.159656 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:32.160492 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:32.160496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:32.160499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:32.160513 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:32.160538 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:32.160552 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:32.160641 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:32.160840 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 19:04:32.160872 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:04:32.163968 24356 catalog_manager.cc:1347] Generated new cluster ID: b4acd9dc4e5044059b5f8e199b6bd4a8
I20260620 19:04:32.164026 24356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:32.175022 24356 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:32.175925 24356 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:32.187433 24356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f88d3d2be748bfa26a3d6655ffc433: Generated new TSK 0
I20260620 19:04:32.188129 24356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:32.196566 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24269
W20260620 19:04:32.201982 24254 connection.cc:535] client connection to 127.0.0.1:32925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32925): Connection reset by peer (error 104)
2026-06-20T19:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:32 dist-test-slave-tsm4 krb5kdc[24368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:32 dist-test-slave-tsm4 krb5kdc[24368](info): set up 2 sockets
Jun 20 19:04:32 dist-test-slave-tsm4 krb5kdc[24368](info): commencing operation
krb5kdc: starting...
W20260620 19:04:33.215425 2882 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:33 dist-test-slave-tsm4 krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:04:33.331689 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:33.445245 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:33.445488 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:33.445528 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:33.449774 24381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:33.449828 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:33.449848 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:33.449891 24381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:33.449925 24381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:33.454943 24381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:33.455937 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:33.456974 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:33.464910 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:33.464949 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:33.464949 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:33.465006 24381 server_base.cc:1034] running on GCE node
I20260620 19:04:33.465788 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:33.466754 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:33.467957 24381 hybrid_clock.cc:648] HybridClock initialized: now 1781982273467942 us; error 24 us; skew 500 ppm
Jun 20 19:04:33 dist-test-slave-tsm4 krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:33.476006 24381 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:04:33.477007 24381 webserver.cc:460] Webserver started at http://127.0.0.1:39505/ using document root <none> and password file <none>
I20260620 19:04:33.477596 24381 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:33.477687 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:33.477953 24381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:33.480161 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "bca6f9723fe44ebcb71fe2b52813347b"
format_stamp: "Formatted at 2026-06-20 19:04:33 on dist-test-slave-tsm4"
server_key: "4c5968da3c1075b07208dfdaee1b5e5d"
server_key_iv: "4cc81d8d0fea23c5da73ffbe6605b4ec"
server_key_version: "encryptionkey@0"
I20260620 19:04:33.480734 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "bca6f9723fe44ebcb71fe2b52813347b"
format_stamp: "Formatted at 2026-06-20 19:04:33 on dist-test-slave-tsm4"
server_key: "4c5968da3c1075b07208dfdaee1b5e5d"
server_key_iv: "4cc81d8d0fea23c5da73ffbe6605b4ec"
server_key_version: "encryptionkey@0"
I20260620 19:04:33.484517 24381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 19:04:33.486989 24396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:33.488322 24381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:33.488471 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "bca6f9723fe44ebcb71fe2b52813347b"
format_stamp: "Formatted at 2026-06-20 19:04:33 on dist-test-slave-tsm4"
server_key: "4c5968da3c1075b07208dfdaee1b5e5d"
server_key_iv: "4cc81d8d0fea23c5da73ffbe6605b4ec"
server_key_version: "encryptionkey@0"
I20260620 19:04:33.488586 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:33.493134 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:33.496206 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:33.496387 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:33.503077 24381 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32821
I20260620 19:04:33.503096 24448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32821 every 8 connection(s)
I20260620 19:04:33.504046 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:33.506619 24449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:33.507273 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24381
I20260620 19:04:33.507448 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:33.507745 2882 external_mini_cluster.cc:1428] Setting key 667342f0163a5f9a5822f5f0c4317477
I20260620 19:04:33.512683 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:33.509059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:50679 (local address 127.0.0.1:32821)
0620 19:04:33.509484 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:33.509521 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:33.509564 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:04:33.510204 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:33.510219 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:33.510482 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:33.510814 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:33.511565 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:33.511569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:33.511581 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:33.511591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:33.511639 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:33.511657 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:33.511801 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:33.511869 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:04:33.512017 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260620 19:04:33.512768 24449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b: Bootstrap starting.
I20260620 19:04:33.515231 24449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:33.515425 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:33.516001 24449 log.cc:824] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:33.518005 24449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b: No bootstrap required, opened a new log
I20260620 19:04:33.519083 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:33.517367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:53223 (local address 127.0.0.1:32821)
0620 19:04:33.517490 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:33.517494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:33.517680 (+ 186us) server_negotiation.cc:407] Connection header received
0620 19:04:33.517789 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:33.517791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:33.517817 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:33.517877 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:33.518665 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:33.518675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:33.518677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:33.518687 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:33.518729 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:33.518748 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:33.518827 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:33.518933 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:04:33.518961 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:04:33.521855 24449 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260620 19:04:33.522081 24449 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:33.522136 24449 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca6f9723fe44ebcb71fe2b52813347b, State: Initialized, Role: FOLLOWER
I20260620 19:04:33.522523 24449 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260620 19:04:33.522666 24449 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:33.522732 24449 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:33.522830 24449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:33.523748 24449 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260620 19:04:33.524072 24449 leader_election.cc:302] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca6f9723fe44ebcb71fe2b52813347b; no voters:
I20260620 19:04:33.524389 24449 leader_election.cc:288] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:33.524474 24459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:33.524720 24459 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [term 1 LEADER]: Becoming Leader. State: Replica: bca6f9723fe44ebcb71fe2b52813347b, State: Running, Role: LEADER
I20260620 19:04:33.525028 24459 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } }
I20260620 19:04:33.525564 24449 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:33.526671 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca6f9723fe44ebcb71fe2b52813347b. Latest consensus state: current_term: 1 leader_uuid: "bca6f9723fe44ebcb71fe2b52813347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } } }
I20260620 19:04:33.526804 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:33.526988 24460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca6f9723fe44ebcb71fe2b52813347b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca6f9723fe44ebcb71fe2b52813347b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32821 } } }
I20260620 19:04:33.527326 24460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:33.527439 24470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:33.530122 24470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:33.535010 24470 catalog_manager.cc:1347] Generated new cluster ID: 4004d6d8786449208a93490ccd88adb8
I20260620 19:04:33.535072 24470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:33.550040 24470 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:33.550345 24470 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:33.561721 24470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca6f9723fe44ebcb71fe2b52813347b: Generated new TSK 0
I20260620 19:04:33.562454 24470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:33.566334 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24381
2026-06-20T19:04:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:33 dist-test-slave-tsm4 krb5kdc[24480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:33 dist-test-slave-tsm4 krb5kdc[24480](info): set up 2 sockets
Jun 20 19:04:33 dist-test-slave-tsm4 krb5kdc[24480](info): commencing operation
krb5kdc: starting...
W20260620 19:04:34.587651 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:34.704461 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:42531
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:34.813566 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:34.813834 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:34.813915 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:34.817785 24493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:34.817862 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:34.817898 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:34.817970 24493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:34.818017 24493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:34.823171 24493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:42531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:34.824170 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:34.825204 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:34.833380 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:34.833443 24493 server_base.cc:1034] running on GCE node
W20260620 19:04:34.833478 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:34.833448 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:34.834123 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:34.835109 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:34.836309 24493 hybrid_clock.cc:648] HybridClock initialized: now 1781982274836293 us; error 28 us; skew 500 ppm
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982274, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:34.838860 24493 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:34.839706 24493 webserver.cc:460] Webserver started at http://127.2.208.190:35889/ using document root <none> and password file <none>
I20260620 19:04:34.840196 24493 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:34.840250 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:34.840408 24493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:34.842242 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "95520836d3184033af78c3aa5de9985b"
format_stamp: "Formatted at 2026-06-20 19:04:34 on dist-test-slave-tsm4"
server_key: "8008f5827af04866d482598be98a80d3"
server_key_iv: "8258735bc461fae6bf934d04e7a9e588"
server_key_version: "encryptionkey@0"
I20260620 19:04:34.842722 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "95520836d3184033af78c3aa5de9985b"
format_stamp: "Formatted at 2026-06-20 19:04:34 on dist-test-slave-tsm4"
server_key: "8008f5827af04866d482598be98a80d3"
server_key_iv: "8258735bc461fae6bf934d04e7a9e588"
server_key_version: "encryptionkey@0"
I20260620 19:04:34.846132 24493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:04:34.848351 24508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:34.849470 24493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:34.849560 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "95520836d3184033af78c3aa5de9985b"
format_stamp: "Formatted at 2026-06-20 19:04:34 on dist-test-slave-tsm4"
server_key: "8008f5827af04866d482598be98a80d3"
server_key_iv: "8258735bc461fae6bf934d04e7a9e588"
server_key_version: "encryptionkey@0"
I20260620 19:04:34.849635 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:34.871142 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:34.873840 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:34.873984 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:34.880959 24493 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:42531
I20260620 19:04:34.880972 24560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:42531 every 8 connection(s)
I20260620 19:04:34.881891 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:34.884552 24561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:34.889840 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b: Bootstrap starting.
I20260620 19:04:34.890043 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24493
I20260620 19:04:34.890184 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:34.890517 2882 external_mini_cluster.cc:1428] Setting key aa22dfa850da624cfea873a1c3a0aaf9
I20260620 19:04:34.892521 24561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:34.893362 24561 log.cc:824] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:34.895112 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b: No bootstrap required, opened a new log
I20260620 19:04:34.898340 24561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } }
I20260620 19:04:34.898578 24561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:34.898653 24561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95520836d3184033af78c3aa5de9985b, State: Initialized, Role: FOLLOWER
I20260620 19:04:34.899194 24561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } }
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:34.899410 24566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:04:34.899554 24566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } }
I20260620 19:04:34.899382 24561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:34.899895 24566 leader_election.cc:302] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95520836d3184033af78c3aa5de9985b; no voters:
I20260620 19:04:34.899952 24561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:34.900095 24561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:34.900285 24566 leader_election.cc:288] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260620 19:04:34.901284 24561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } }
I20260620 19:04:34.901428 24561 leader_election.cc:302] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95520836d3184033af78c3aa5de9985b; no voters:
I20260620 19:04:34.901484 24567 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 19:04:34.901523 24561 leader_election.cc:288] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:34.901746 24566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:34.901980 24566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [term 1 LEADER]: Becoming Leader. State: Replica: 95520836d3184033af78c3aa5de9985b, State: Running, Role: LEADER
I20260620 19:04:34.902338 24566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } }
I20260620 19:04:34.902696 24561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:34.903779 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95520836d3184033af78c3aa5de9985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } } }
I20260620 19:04:34.903784 24567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95520836d3184033af78c3aa5de9985b. Latest consensus state: current_term: 1 leader_uuid: "95520836d3184033af78c3aa5de9985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95520836d3184033af78c3aa5de9985b" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 42531 } } }
I20260620 19:04:34.903976 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:34.903976 24567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:34.904392 24576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:34.906355 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:34.891844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49017 (local address 127.2.208.190:42531)
0620 19:04:34.892234 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:34.892252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:34.892293 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:04:34.893112 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:34.893133 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:34.893549 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:34.893933 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:34.894710 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:34.895757 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:34.896622 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:34.897090 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:34.897254 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:34.900141 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:34.900149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:34.900166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:34.900199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:34.902844 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:34.903481 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:34.903490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:34.903498 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:34.903653 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:34.904097 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:34.904101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:34.904103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:34.904644 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:34.904863 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:34.905189 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 19:04:34.905433 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260620 19:04:34.907509 24576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:34.909914 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:34.913234 24576 catalog_manager.cc:1347] Generated new cluster ID: 0361254fc35046a2ab61ef94452252e7
I20260620 19:04:34.913290 24576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:34.919896 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:34.912318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41797 (local address 127.2.208.190:42531)
0620 19:04:34.912429 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:34.912433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:34.912722 (+ 289us) server_negotiation.cc:407] Connection header received
0620 19:04:34.912835 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:34.912838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:34.912881 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:34.912965 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:34.913698 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:34.914213 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:34.914993 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:34.915278 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:34.915403 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:34.916750 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:34.916754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:34.916756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:34.916778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:34.918166 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:34.918627 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:34.918631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:34.918632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:34.918714 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:34.919030 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:34.919033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:34.919035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:34.919192 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:34.919341 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:34.919613 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:04:34.919699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:04:34.929517 24576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:34.930389 24576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:34.939054 24576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95520836d3184033af78c3aa5de9985b: Generated new TSK 0
I20260620 19:04:34.939771 24576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:34.946393 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24493
W20260620 19:04:34.951962 24478 connection.cc:535] client connection to 127.2.208.190:42531 recv error: Network error: failed to read from TLS socket (remote: 127.2.208.190:42531): Connection reset by peer (error 104)
2026-06-20T19:04:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24593](info): set up 2 sockets
Jun 20 19:04:34 dist-test-slave-tsm4 krb5kdc[24593](info): commencing operation
krb5kdc: starting...
W20260620 19:04:35.975682 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:36 dist-test-slave-tsm4 krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:36.092410 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:40879
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:40879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:36.200387 24606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:36.200663 24606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:36.200754 24606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:36.204723 24606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:36.204797 24606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:36.204833 24606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:36.204890 24606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:36.204926 24606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:36.210053 24606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:40879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:36.211062 24606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:36.212093 24606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:36.220297 24614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:36.220256 24611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:36.220258 24612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:36.220621 24606 server_base.cc:1034] running on GCE node
I20260620 19:04:36.221158 24606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:36.222196 24606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:36.223397 24606 hybrid_clock.cc:648] HybridClock initialized: now 1781982276223350 us; error 28 us; skew 500 ppm
Jun 20 19:04:36 dist-test-slave-tsm4 krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982276, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:36.226145 24606 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:36.226979 24606 webserver.cc:460] Webserver started at http://127.2.208.190:46557/ using document root <none> and password file <none>
I20260620 19:04:36.227541 24606 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:36.227605 24606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:36.227816 24606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:36.229744 24606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "ae3c63638f8c438e9faec2a1cf6d83d5"
format_stamp: "Formatted at 2026-06-20 19:04:36 on dist-test-slave-tsm4"
server_key: "cc457c1bf22a89a3fe23104d1ca0a320"
server_key_iv: "4f832ffe64a6565805f741ee4b798d69"
server_key_version: "encryptionkey@0"
I20260620 19:04:36.230230 24606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "ae3c63638f8c438e9faec2a1cf6d83d5"
format_stamp: "Formatted at 2026-06-20 19:04:36 on dist-test-slave-tsm4"
server_key: "cc457c1bf22a89a3fe23104d1ca0a320"
server_key_iv: "4f832ffe64a6565805f741ee4b798d69"
server_key_version: "encryptionkey@0"
I20260620 19:04:36.233718 24606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:04:36.235954 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:36.237113 24606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 19:04:36.237236 24606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "ae3c63638f8c438e9faec2a1cf6d83d5"
format_stamp: "Formatted at 2026-06-20 19:04:36 on dist-test-slave-tsm4"
server_key: "cc457c1bf22a89a3fe23104d1ca0a320"
server_key_iv: "4f832ffe64a6565805f741ee4b798d69"
server_key_version: "encryptionkey@0"
I20260620 19:04:36.237344 24606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:36.266340 24606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:36.269781 24606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:36.269994 24606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:36.277132 24606 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:40879
I20260620 19:04:36.277151 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:40879 every 8 connection(s)
I20260620 19:04:36.278054 24606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:36.278724 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24606
I20260620 19:04:36.278884 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:36.279160 2882 external_mini_cluster.cc:1428] Setting key e66f5631d800a389d4093a67368a890a
I20260620 19:04:36.281203 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:36.286628 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5: Bootstrap starting.
I20260620 19:04:36.288053 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:36.280579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45483 (local address 127.2.208.190:40879)
0620 19:04:36.281027 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:36.281042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:36.281076 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:36.281768 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:36.281786 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:36.282115 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:36.282449 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:36.282838 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:36.283896 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:36.284788 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:36.285269 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:36.285446 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:36.286353 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:36.286358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:36.286370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:36.286732 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:36.286819 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:36.286851 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:36.286986 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:36.287168 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:04:36.287303 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":131,"threads_started":1}
I20260620 19:04:36.289048 24674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:36.289822 24674 log.cc:824] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:36.290839 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:36.291720 24674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5: No bootstrap required, opened a new log
I20260620 19:04:36.295818 24674 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } }
I20260620 19:04:36.296119 24674 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:36.296233 24674 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3c63638f8c438e9faec2a1cf6d83d5, State: Initialized, Role: FOLLOWER
I20260620 19:04:36.296706 24674 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } }
I20260620 19:04:36.296869 24674 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:36.296978 24674 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:36.297015 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:36.292878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57971 (local address 127.2.208.190:40879)
0620 19:04:36.292999 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:36.293003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:36.293185 (+ 182us) server_negotiation.cc:407] Connection header received
0620 19:04:36.293295 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:36.293297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:36.293333 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:36.293394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:36.293722 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:36.294200 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:36.295305 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:36.295599 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:36.295674 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:36.296539 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:36.296542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:36.296545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:36.296557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:36.296585 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:36.296601 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:36.296666 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:36.296851 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:04:36.296888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:04:36.297082 24674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:36.297938 24674 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } }
I20260620 19:04:36.298261 24674 leader_election.cc:302] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae3c63638f8c438e9faec2a1cf6d83d5; no voters:
I20260620 19:04:36.298540 24674 leader_election.cc:288] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:36.298692 24684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:36.298949 24684 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [term 1 LEADER]: Becoming Leader. State: Replica: ae3c63638f8c438e9faec2a1cf6d83d5, State: Running, Role: LEADER
I20260620 19:04:36.299276 24684 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } }
I20260620 19:04:36.299777 24674 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:36.300927 24686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae3c63638f8c438e9faec2a1cf6d83d5. Latest consensus state: current_term: 1 leader_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } } }
I20260620 19:04:36.300967 24685 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3c63638f8c438e9faec2a1cf6d83d5" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 40879 } } }
I20260620 19:04:36.301062 24685 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:36.301062 24686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:36.301457 24693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:36.304665 24693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:36.310425 24693 catalog_manager.cc:1347] Generated new cluster ID: 03efa62f5b374e8eafd666766779cb51
I20260620 19:04:36.310505 24693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:36.315725 24693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:36.316555 24693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:36.329713 24693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae3c63638f8c438e9faec2a1cf6d83d5: Generated new TSK 0
I20260620 19:04:36.330399 24693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:36.340526 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24606
W20260620 19:04:36.346107 24591 connection.cc:535] client connection to 127.2.208.190:40879 recv error: Network error: failed to read from TLS socket (remote: 127.2.208.190:40879): Connection reset by peer (error 104)
2026-06-20T19:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:36 dist-test-slave-tsm4 krb5kdc[24705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:36 dist-test-slave-tsm4 krb5kdc[24705](info): set up 2 sockets
Jun 20 19:04:36 dist-test-slave-tsm4 krb5kdc[24705](info): commencing operation
krb5kdc: starting...
W20260620 19:04:37.375564 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:37 dist-test-slave-tsm4 krb5kdc[24705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:37.494207 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34881
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:37.601830 24718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:37.602109 24718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:37.602183 24718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:37.606076 24718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:37.606127 24718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:37.606145 24718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:37.606197 24718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:37.606227 24718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:37.611411 24718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:34881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:37.612399 24718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:37.613420 24718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:37.621601 24723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:37.621629 24726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:37.621635 24724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:37.621672 24718 server_base.cc:1034] running on GCE node
I20260620 19:04:37.622403 24718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:37.623493 24718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:37.624682 24718 hybrid_clock.cc:648] HybridClock initialized: now 1781982277624670 us; error 36 us; skew 500 ppm
Jun 20 19:04:37 dist-test-slave-tsm4 krb5kdc[24705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982277, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:37.627264 24718 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:37.628130 24718 webserver.cc:460] Webserver started at http://127.2.208.190:34733/ using document root <none> and password file <none>
I20260620 19:04:37.628681 24718 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:37.628786 24718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:37.629007 24718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:37.631053 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "ca0621a832984e249f5e2d394d9088a6"
format_stamp: "Formatted at 2026-06-20 19:04:37 on dist-test-slave-tsm4"
server_key: "5d3b0cde5b1aa0e5ef9a10f612abd5ef"
server_key_iv: "f7c52de88a2578eda56b36b62cec2c6a"
server_key_version: "encryptionkey@0"
I20260620 19:04:37.631618 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "ca0621a832984e249f5e2d394d9088a6"
format_stamp: "Formatted at 2026-06-20 19:04:37 on dist-test-slave-tsm4"
server_key: "5d3b0cde5b1aa0e5ef9a10f612abd5ef"
server_key_iv: "f7c52de88a2578eda56b36b62cec2c6a"
server_key_version: "encryptionkey@0"
I20260620 19:04:37.635098 24718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:04:37.637418 24733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:37.638592 24718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:37.638726 24718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "ca0621a832984e249f5e2d394d9088a6"
format_stamp: "Formatted at 2026-06-20 19:04:37 on dist-test-slave-tsm4"
server_key: "5d3b0cde5b1aa0e5ef9a10f612abd5ef"
server_key_iv: "f7c52de88a2578eda56b36b62cec2c6a"
server_key_version: "encryptionkey@0"
I20260620 19:04:37.638841 24718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:37.643191 24718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:37.645851 24718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:37.646029 24718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:37.652976 24718 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:34881
I20260620 19:04:37.653002 24785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:34881 every 8 connection(s)
I20260620 19:04:37.653839 24718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:37.656445 24786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:37.659739 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24718
I20260620 19:04:37.659885 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:37.660162 2882 external_mini_cluster.cc:1428] Setting key 771126f471308acfc5b03adc3881ffc5
I20260620 19:04:37.662230 24786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6: Bootstrap starting.
I20260620 19:04:37.664572 24786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:37.665225 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:37.661564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45423 (local address 127.2.208.190:34881)
0620 19:04:37.661970 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:37.661984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:37.662018 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:37.662668 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:37.662686 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:37.662974 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:37.663282 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:37.664084 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:37.664088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:37.664100 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:37.664109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:37.664159 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:37.664177 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:37.664292 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:37.664391 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:04:37.664542 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260620 19:04:37.665366 24786 log.cc:824] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:37.667352 24786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6: No bootstrap required, opened a new log
I20260620 19:04:37.667840 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:37.671451 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:37.669780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40289 (local address 127.2.208.190:34881)
0620 19:04:37.669886 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:37.669891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:37.670044 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:04:37.670168 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:37.670170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:37.670195 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:37.670262 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:37.670992 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:37.670994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:37.670996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:37.671006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:37.671028 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:37.671037 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:37.671114 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:37.671231 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:04:37.671266 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:04:37.671507 24786 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } }
I20260620 19:04:37.671792 24786 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:37.671867 24786 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0621a832984e249f5e2d394d9088a6, State: Initialized, Role: FOLLOWER
I20260620 19:04:37.672288 24786 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } }
I20260620 19:04:37.672447 24786 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:37.672523 24786 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:37.672616 24786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:37.673447 24786 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } }
I20260620 19:04:37.673774 24786 leader_election.cc:302] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0621a832984e249f5e2d394d9088a6; no voters:
I20260620 19:04:37.674054 24786 leader_election.cc:288] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:37.674163 24796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:37.674428 24796 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [term 1 LEADER]: Becoming Leader. State: Replica: ca0621a832984e249f5e2d394d9088a6, State: Running, Role: LEADER
I20260620 19:04:37.674810 24796 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } }
I20260620 19:04:37.675225 24786 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:37.676321 24797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0621a832984e249f5e2d394d9088a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } } }
I20260620 19:04:37.676463 24797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:37.676775 24798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0621a832984e249f5e2d394d9088a6. Latest consensus state: current_term: 1 leader_uuid: "ca0621a832984e249f5e2d394d9088a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0621a832984e249f5e2d394d9088a6" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 34881 } } }
I20260620 19:04:37.676856 24798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:37.676862 24805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:37.679626 24805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:37.684517 24805 catalog_manager.cc:1347] Generated new cluster ID: 574b5f9e4ffa44c3ba6e639bc3b4f34b
I20260620 19:04:37.684579 24805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:37.706110 24805 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:37.706459 24805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:37.715299 24805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0621a832984e249f5e2d394d9088a6: Generated new TSK 0
I20260620 19:04:37.715984 24805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:37.726428 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24718
2026-06-20T19:04:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:37 dist-test-slave-tsm4 krb5kdc[24817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:37 dist-test-slave-tsm4 krb5kdc[24817](info): set up 2 sockets
Jun 20 19:04:37 dist-test-slave-tsm4 krb5kdc[24817](info): commencing operation
krb5kdc: starting...
W20260620 19:04:38.759756 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:38 dist-test-slave-tsm4 krb5kdc[24817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:38.877898 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41135
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:38.985334 24830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:38.985632 24830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:38.985674 24830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:38.989590 24830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:38.989650 24830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:38.989670 24830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:38.989686 24830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:38.989701 24830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:38.994860 24830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:38.995934 24830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:38.996906 24830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:39.005064 24836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:39.005206 24838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:39.005088 24835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:39.005347 24830 server_base.cc:1034] running on GCE node
I20260620 19:04:39.006001 24830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:39.007042 24830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:39.008201 24830 hybrid_clock.cc:648] HybridClock initialized: now 1781982279008187 us; error 25 us; skew 500 ppm
Jun 20 19:04:39 dist-test-slave-tsm4 krb5kdc[24817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982279, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:39.013787 24830 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:39.014703 24830 webserver.cc:460] Webserver started at http://127.2.208.190:46585/ using document root <none> and password file <none>
I20260620 19:04:39.015267 24830 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:39.015326 24830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:39.015605 24830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:39.017581 24830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "417792bc10ec433eb07340f2ca5f6b70"
format_stamp: "Formatted at 2026-06-20 19:04:39 on dist-test-slave-tsm4"
server_key: "b952260235016caf4f8ebc31a035f551"
server_key_iv: "fd9a7a9241789e7afba04304266247a5"
server_key_version: "encryptionkey@0"
I20260620 19:04:39.018116 24830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "417792bc10ec433eb07340f2ca5f6b70"
format_stamp: "Formatted at 2026-06-20 19:04:39 on dist-test-slave-tsm4"
server_key: "b952260235016caf4f8ebc31a035f551"
server_key_iv: "fd9a7a9241789e7afba04304266247a5"
server_key_version: "encryptionkey@0"
I20260620 19:04:39.021754 24830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:04:39.024200 24845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:39.025445 24830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:39.025571 24830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "417792bc10ec433eb07340f2ca5f6b70"
format_stamp: "Formatted at 2026-06-20 19:04:39 on dist-test-slave-tsm4"
server_key: "b952260235016caf4f8ebc31a035f551"
server_key_iv: "fd9a7a9241789e7afba04304266247a5"
server_key_version: "encryptionkey@0"
I20260620 19:04:39.025671 24830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:39.043325 24830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:39.046168 24830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:39.046406 24830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:39.053370 24830 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:41135
I20260620 19:04:39.053391 24897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:41135 every 8 connection(s)
I20260620 19:04:39.054322 24830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:39.057164 24898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:39.062660 24898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70: Bootstrap starting.
I20260620 19:04:39.063934 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24830
I20260620 19:04:39.064078 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:39.064378 2882 external_mini_cluster.cc:1428] Setting key 93780c281f2b468565a4961b8a1fdf7b
I20260620 19:04:39.065701 24898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:39.066454 24898 log.cc:824] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:39.068619 24898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70: No bootstrap required, opened a new log
I20260620 19:04:39.073012 24898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } }
I20260620 19:04:39.073266 24898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 20 19:04:39 dist-test-slave-tsm4 krb5kdc[24817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:39.073350 24898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417792bc10ec433eb07340f2ca5f6b70, State: Initialized, Role: FOLLOWER
I20260620 19:04:39.073764 24898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } }
I20260620 19:04:39.073916 24898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:39.073985 24898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:39.074106 24898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:39.074965 24898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } }
I20260620 19:04:39.075292 24898 leader_election.cc:302] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 417792bc10ec433eb07340f2ca5f6b70; no voters:
I20260620 19:04:39.075624 24898 leader_election.cc:288] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:39.075690 24903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:39.075884 24903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [term 1 LEADER]: Becoming Leader. State: Replica: 417792bc10ec433eb07340f2ca5f6b70, State: Running, Role: LEADER
I20260620 19:04:39.076195 24903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } }
I20260620 19:04:39.076917 24898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:39.077787 24904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417792bc10ec433eb07340f2ca5f6b70. Latest consensus state: current_term: 1 leader_uuid: "417792bc10ec433eb07340f2ca5f6b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } } }
I20260620 19:04:39.077924 24904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:39.078311 24912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:39.080099 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:39.065485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40893 (local address 127.2.208.190:41135)
0620 19:04:39.065889 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:39.065904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:39.065938 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:39.066586 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:39.066611 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:39.067040 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:39.067404 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:39.068240 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:39.069283 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:39.070446 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:39.070896 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:39.071007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:39.073954 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:39.073959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:39.073973 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:39.074025 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:39.076082 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:39.076623 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:39.076629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:39.076635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:39.076758 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:39.077058 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:39.077063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:39.077065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:39.077657 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:39.077864 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:39.078717 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 19:04:39.079023 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260620 19:04:39.080864 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "417792bc10ec433eb07340f2ca5f6b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417792bc10ec433eb07340f2ca5f6b70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 41135 } } }
I20260620 19:04:39.080958 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:39.083488 24912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:39.083686 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:39.089270 24912 catalog_manager.cc:1347] Generated new cluster ID: b6f81940b29b4bbd8db75e3a9df8221c
I20260620 19:04:39.089335 24912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:39.094944 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:39.085909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43953 (local address 127.2.208.190:41135)
0620 19:04:39.086009 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:39.086013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:39.086245 (+ 232us) server_negotiation.cc:407] Connection header received
0620 19:04:39.086344 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:39.086347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:39.086396 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:39.086479 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:39.087272 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:39.088161 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:39.089389 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:39.089745 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:39.089853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:39.091480 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:39.091485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:39.091487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:39.091513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:39.093202 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:39.093669 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:39.093673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:39.093675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:39.093762 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:39.094054 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:39.094057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:39.094059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:39.094287 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:39.094429 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:39.094666 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:04:39.094779 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:04:39.096601 24912 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:39.097450 24912 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:39.108299 24912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 417792bc10ec433eb07340f2ca5f6b70: Generated new TSK 0
I20260620 19:04:39.108978 24912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:39.113382 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24830
2026-06-20T19:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:39 dist-test-slave-tsm4 krb5kdc[24929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:39 dist-test-slave-tsm4 krb5kdc[24929](info): set up 2 sockets
Jun 20 19:04:39 dist-test-slave-tsm4 krb5kdc[24929](info): commencing operation
krb5kdc: starting...
W20260620 19:04:40.167567 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:40 dist-test-slave-tsm4 krb5kdc[24929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:40.284134 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:32919
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:32919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:40.402849 24942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:40.403121 24942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:40.403167 24942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:40.407094 24942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:40.407152 24942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:40.407171 24942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:40.407187 24942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:40.407202 24942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:40.412397 24942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:32919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:32919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:40.413506 24942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:40.414760 24942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:40.423120 24948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:40.423238 24950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:40.423120 24947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:40.423429 24942 server_base.cc:1034] running on GCE node
I20260620 19:04:40.424113 24942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:40.425226 24942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:40.426414 24942 hybrid_clock.cc:648] HybridClock initialized: now 1781982280426406 us; error 29 us; skew 500 ppm
Jun 20 19:04:40 dist-test-slave-tsm4 krb5kdc[24929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982280, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:40.429272 24942 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:40.430186 24942 webserver.cc:460] Webserver started at http://127.2.208.190:35171/ using document root <none> and password file <none>
I20260620 19:04:40.430745 24942 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:40.430833 24942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:40.431077 24942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:40.433122 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "8e8da7c84be044efb23c01296feac556"
format_stamp: "Formatted at 2026-06-20 19:04:40 on dist-test-slave-tsm4"
server_key: "9271475287af44bbb70d0cbd94915f1c"
server_key_iv: "0fe169aef52d53864ce0116e49975ed5"
server_key_version: "encryptionkey@0"
I20260620 19:04:40.433686 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "8e8da7c84be044efb23c01296feac556"
format_stamp: "Formatted at 2026-06-20 19:04:40 on dist-test-slave-tsm4"
server_key: "9271475287af44bbb70d0cbd94915f1c"
server_key_iv: "0fe169aef52d53864ce0116e49975ed5"
server_key_version: "encryptionkey@0"
I20260620 19:04:40.437492 24942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:04:40.440657 24957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:40.441843 24942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:40.441941 24942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "8e8da7c84be044efb23c01296feac556"
format_stamp: "Formatted at 2026-06-20 19:04:40 on dist-test-slave-tsm4"
server_key: "9271475287af44bbb70d0cbd94915f1c"
server_key_iv: "0fe169aef52d53864ce0116e49975ed5"
server_key_version: "encryptionkey@0"
I20260620 19:04:40.442080 24942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:40.463688 24942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:40.466912 24942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:40.467083 24942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:40.474198 24942 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:32919
I20260620 19:04:40.474217 25009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:32919 every 8 connection(s)
I20260620 19:04:40.475132 24942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:40.477820 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:40.480048 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 24942
I20260620 19:04:40.480189 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:40.480489 2882 external_mini_cluster.cc:1428] Setting key b85b6d78ad856e919d272697bebb7536
I20260620 19:04:40.483978 25010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556: Bootstrap starting.
I20260620 19:04:40.486392 25010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:40.487239 25010 log.cc:824] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:40.489239 25010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556: No bootstrap required, opened a new log
I20260620 19:04:40.489279 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:40.481780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52581 (local address 127.2.208.190:32919)
0620 19:04:40.482187 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:40.482202 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:40.482234 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:04:40.483018 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:40.483035 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:40.483443 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:40.483714 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:40.484077 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:40.485040 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:40.486173 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:40.486670 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:40.486781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:40.487575 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:40.487580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:40.487591 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:40.488098 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:40.488147 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:40.488168 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:40.488241 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:40.488396 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:04:40.488624 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":144,"threads_started":1}
I20260620 19:04:40.492146 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:40.493454 25010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } }
I20260620 19:04:40.493729 25010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:40.493789 25010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8da7c84be044efb23c01296feac556, State: Initialized, Role: FOLLOWER
I20260620 19:04:40.494246 25010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } }
I20260620 19:04:40.494432 25010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:40.494493 25010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:40.494618 25010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:40.495764 25010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } }
I20260620 19:04:40.496093 25010 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8da7c84be044efb23c01296feac556; no voters:
I20260620 19:04:40.496404 25010 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:40.496480 25020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:40.496658 25020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [term 1 LEADER]: Becoming Leader. State: Replica: 8e8da7c84be044efb23c01296feac556, State: Running, Role: LEADER
I20260620 19:04:40.496946 25020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } }
I20260620 19:04:40.497646 25010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:40.498096 25020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8da7c84be044efb23c01296feac556. Latest consensus state: current_term: 1 leader_uuid: "8e8da7c84be044efb23c01296feac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } } }
I20260620 19:04:40.498200 25020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:40.498801 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:40.494332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43351 (local address 127.2.208.190:32919)
0620 19:04:40.494441 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:40.494445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:40.494578 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:04:40.494768 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:40.494771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:40.494841 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:40.494920 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:40.495218 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:40.495871 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:40.496927 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:40.497250 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:40.497308 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:40.498325 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:40.498329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:40.498332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:40.498347 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:40.498372 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:40.498384 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:40.498432 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:40.498603 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:04:40.498664 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:04:40.498868 25021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8da7c84be044efb23c01296feac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8da7c84be044efb23c01296feac556" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 32919 } } }
I20260620 19:04:40.498958 25021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:40.499209 25024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:40.502617 25024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:40.507725 25024 catalog_manager.cc:1347] Generated new cluster ID: c18b83e6b51649f7a117a5d6bd6e4390
I20260620 19:04:40.507786 25024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:40.524489 25024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:40.525722 25024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:40.539225 25024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8da7c84be044efb23c01296feac556: Generated new TSK 0
I20260620 19:04:40.539916 25024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:40.549420 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 24942
2026-06-20T19:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:40 dist-test-slave-tsm4 krb5kdc[25041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:40 dist-test-slave-tsm4 krb5kdc[25041](info): set up 2 sockets
Jun 20 19:04:40 dist-test-slave-tsm4 krb5kdc[25041](info): commencing operation
krb5kdc: starting...
W20260620 19:04:41.589258 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:41 dist-test-slave-tsm4 krb5kdc[25041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:41.720139 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38349
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:41.828287 25054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:41.828540 25054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:41.828580 25054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:41.832659 25054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:41.832715 25054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:41.832734 25054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:41.832751 25054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:41.832790 25054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:41.837867 25054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:41.838871 25054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:41.839906 25054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:41.848055 25062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:41.848045 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:41.848043 25060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:41.848336 25054 server_base.cc:1034] running on GCE node
I20260620 19:04:41.848964 25054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:41.850035 25054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:41.851192 25054 hybrid_clock.cc:648] HybridClock initialized: now 1781982281851164 us; error 40 us; skew 500 ppm
Jun 20 19:04:41 dist-test-slave-tsm4 krb5kdc[25041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982281, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:41.853888 25054 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:41.854766 25054 webserver.cc:460] Webserver started at http://127.2.208.190:33833/ using document root <none> and password file <none>
I20260620 19:04:41.855301 25054 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:41.855413 25054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:41.855636 25054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:41.857604 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "e8dcecc33caa4b989f5c6dea01ebbd70"
format_stamp: "Formatted at 2026-06-20 19:04:41 on dist-test-slave-tsm4"
server_key: "fd2931dfa38b7c4f26ac41f189deb2f7"
server_key_iv: "b52a97fb124dfde5dfb199262b281f3f"
server_key_version: "encryptionkey@0"
I20260620 19:04:41.858175 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "e8dcecc33caa4b989f5c6dea01ebbd70"
format_stamp: "Formatted at 2026-06-20 19:04:41 on dist-test-slave-tsm4"
server_key: "fd2931dfa38b7c4f26ac41f189deb2f7"
server_key_iv: "b52a97fb124dfde5dfb199262b281f3f"
server_key_version: "encryptionkey@0"
I20260620 19:04:41.861717 25054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:04:41.864110 25069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:41.865311 25054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:41.865453 25054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "e8dcecc33caa4b989f5c6dea01ebbd70"
format_stamp: "Formatted at 2026-06-20 19:04:41 on dist-test-slave-tsm4"
server_key: "fd2931dfa38b7c4f26ac41f189deb2f7"
server_key_iv: "b52a97fb124dfde5dfb199262b281f3f"
server_key_version: "encryptionkey@0"
I20260620 19:04:41.865562 25054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:41.870093 25054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:41.873263 25054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:41.873445 25054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:41.880273 25054 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:38349
I20260620 19:04:41.880286 25121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:38349 every 8 connection(s)
I20260620 19:04:41.881168 25054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:41.883912 25122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:41.885525 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25054
I20260620 19:04:41.885663 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:41.885953 2882 external_mini_cluster.cc:1428] Setting key d7031bf589a156650c866bdba3f498dd
I20260620 19:04:41.890525 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70: Bootstrap starting.
I20260620 19:04:41.890964 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:41.887210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40597 (local address 127.2.208.190:38349)
0620 19:04:41.887750 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:41.887765 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:41.887799 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:41.888395 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:41.888413 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:41.888673 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:41.888946 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:41.889751 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:41.889756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:41.889768 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:41.889777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:41.889825 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:41.889843 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:41.889962 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:41.890063 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 19:04:41.890223 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260620 19:04:41.892673 25122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:41.893376 25122 log.cc:824] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:41.893584 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:41.897222 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:41.895676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42247 (local address 127.2.208.190:38349)
0620 19:04:41.895807 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:41.895813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:41.896026 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:04:41.896113 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:41.896116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:41.896147 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:41.896219 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:41.896870 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:41.896874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:41.896876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:04:41.896887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:41.896922 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:04:41.896936 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:41.897022 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:41.897081 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:04:41.897115 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:04:41.897360 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70: No bootstrap required, opened a new log
I20260620 19:04:41.900985 25122 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } }
I20260620 19:04:41.901219 25122 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:04:41.901279 25122 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8dcecc33caa4b989f5c6dea01ebbd70, State: Initialized, Role: FOLLOWER
I20260620 19:04:41.901664 25122 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } }
I20260620 19:04:41.901808 25122 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:04:41.901895 25122 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:04:41.902005 25122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:41.902832 25122 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } }
I20260620 19:04:41.903159 25122 leader_election.cc:302] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8dcecc33caa4b989f5c6dea01ebbd70; no voters:
I20260620 19:04:41.903543 25122 leader_election.cc:288] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:04:41.903630 25132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:41.903836 25132 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [term 1 LEADER]: Becoming Leader. State: Replica: e8dcecc33caa4b989f5c6dea01ebbd70, State: Running, Role: LEADER
I20260620 19:04:41.904119 25132 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } }
I20260620 19:04:41.904668 25122 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:41.905823 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } } }
I20260620 19:04:41.905838 25134 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8dcecc33caa4b989f5c6dea01ebbd70. Latest consensus state: current_term: 1 leader_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dcecc33caa4b989f5c6dea01ebbd70" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 38349 } } }
I20260620 19:04:41.905972 25134 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:41.905970 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:41.906927 25141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:41.909590 25141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:41.914789 25141 catalog_manager.cc:1347] Generated new cluster ID: ecc2cac25ed84ddabd292fa840830cc9
I20260620 19:04:41.914846 25141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:41.929009 25141 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:41.929303 25141 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:41.934108 25141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8dcecc33caa4b989f5c6dea01ebbd70: Generated new TSK 0
I20260620 19:04:41.934753 25141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:04:41.948393 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25054
2026-06-20T19:04:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1398 ms)
[----------] 18 tests from AuthTokenIssuingTest (24962 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:42 dist-test-slave-tsm4 krb5kdc[25153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:42 dist-test-slave-tsm4 krb5kdc[25153](info): set up 2 sockets
Jun 20 19:04:42 dist-test-slave-tsm4 krb5kdc[25153](info): commencing operation
krb5kdc: starting...
W20260620 19:04:42.975557 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:43.094782 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:46645
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:43.205313 25166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:43.205578 25166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:43.205600 25166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:43.205677 25166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:43.209741 25166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:43.209816 25166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:43.209837 25166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:43.209854 25166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:43.209869 25166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:43.214890 25166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:43.215960 25166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:43.216930 25166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:43.225366 25171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:43.225654 25174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:43.225705 25166 server_base.cc:1034] running on GCE node
W20260620 19:04:43.225358 25172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:43.226480 25166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:43.227639 25166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:43.228820 25166 hybrid_clock.cc:648] HybridClock initialized: now 1781982283228795 us; error 43 us; skew 500 ppm
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:43.231665 25166 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:43.232528 25166 webserver.cc:460] Webserver started at http://127.2.208.190:34531/ using document root <none> and password file <none>
I20260620 19:04:43.233110 25166 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:43.233201 25166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:43.233428 25166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:43.235498 25166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "bc728e22dd7f43fdb709744473a46887"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "548a5abfacd1426665dfd88d37605064"
server_key_iv: "79368d687a3e46e9c788c069e5028cbc"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.236052 25166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "bc728e22dd7f43fdb709744473a46887"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "548a5abfacd1426665dfd88d37605064"
server_key_iv: "79368d687a3e46e9c788c069e5028cbc"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.239580 25166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:04:43.241973 25181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:43.243089 25166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:43.243222 25166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "bc728e22dd7f43fdb709744473a46887"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "548a5abfacd1426665dfd88d37605064"
server_key_iv: "79368d687a3e46e9c788c069e5028cbc"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.243345 25166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:43.256397 25166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:43.259099 25166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:43.259291 25166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:43.266214 25166 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:46645
I20260620 19:04:43.266496 25233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:46645 every 8 connection(s)
I20260620 19:04:43.267144 25166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:43.269961 25234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:43.270762 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25166
I20260620 19:04:43.270898 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:43.271178 2882 external_mini_cluster.cc:1428] Setting key 7ea0709586fb684c4ff5f2a71d4a7a4e
I20260620 19:04:43.273739 25234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.189@KRBTEST.COM" created.
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:43.288216 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.276103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46645 (local address 127.0.0.1:40258)
0620 19:04:43.276568 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.276606 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.277455 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.278072 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.278076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.278515 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.278796 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.278809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.279980 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.279984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.280683 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.280692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.281411 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.281419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.281567 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.282180 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.282207 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.283607 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.285445 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.285455 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.285466 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.285725 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.286100 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.286102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.286103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.286200 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.286957 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.286961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.287070 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.287197 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.287477 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260620 19:04:43.288225 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.277037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.2.208.190:46645)
0620 19:04:43.277320 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.277331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.277373 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:04:43.277730 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.277734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.277902 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.278078 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.278933 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.279852 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.280801 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.281188 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.281346 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.283752 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.283757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.283765 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.283785 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.285327 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.285840 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.285849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.285852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.285998 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.286294 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.286301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.286304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.286840 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.286985 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.287316 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.287583 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260620 19:04:43.290740 25234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:43.291517 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.291174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41184)
0620 19:04:43.291315 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.291429 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:04:43.291720 25184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
I20260620 19:04:43.292541 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.292362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41190)
0620 19:04:43.292457 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.292497 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:04:43.292682 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:33603: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
I20260620 19:04:43.296402 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:33603
--webserver_interface=127.2.208.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:04:43.341940 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } attempt: 1
I20260620 19:04:43.342871 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.342525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41194)
0620 19:04:43.342708 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.342779 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 19:04:43.343705 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.343473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41202)
0620 19:04:43.343577 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.343640 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:04:43.343833 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:33603: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
W20260620 19:04:43.408958 25241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:43.409222 25241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:43.409245 25241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:43.409282 25241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:04:43.411978 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } attempt: 2
I20260620 19:04:43.412835 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.412515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41204)
0620 19:04:43.412680 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.412747 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 19:04:43.413525 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.413305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41216)
0620 19:04:43.413423 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.413469 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 19:04:43.413672 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:33603: Network error: Client connection negotiation failed: client connection to 127.2.208.189:33603: connect: Connection refused (error 111)
W20260620 19:04:43.413823 25241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:43.413879 25241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:43.413919 25241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:43.413954 25241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:43.413985 25241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:43.419201 25241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:33603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.208.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:43.420187 25241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:43.421177 25241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:43.429183 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:43.429188 25246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:43.429242 25247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:43.429649 25241 server_base.cc:1034] running on GCE node
I20260620 19:04:43.430181 25241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:43.431231 25241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:43.432428 25241 hybrid_clock.cc:648] HybridClock initialized: now 1781982283432404 us; error 43 us; skew 500 ppm
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:43.434993 25241 init.cc:377] Logged in from keytab as kudu/127.2.208.189@KRBTEST.COM (short username kudu)
I20260620 19:04:43.435907 25241 webserver.cc:460] Webserver started at http://127.2.208.189:45467/ using document root <none> and password file <none>
I20260620 19:04:43.436460 25241 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:43.436553 25241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:43.436789 25241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:43.438760 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data/instance:
uuid: "5aa9f392942a42c2b66cd04e67c6a0ec"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "a9fd555b952057a86e77b179cb6b5b14"
server_key_iv: "fbda9195e49aaf5fb4d17e7f050bbf21"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.439311 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal/instance:
uuid: "5aa9f392942a42c2b66cd04e67c6a0ec"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "a9fd555b952057a86e77b179cb6b5b14"
server_key_iv: "fbda9195e49aaf5fb4d17e7f050bbf21"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.442843 25241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:04:43.445261 25256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:43.446506 25241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:43.446651 25241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "5aa9f392942a42c2b66cd04e67c6a0ec"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "a9fd555b952057a86e77b179cb6b5b14"
server_key_iv: "fbda9195e49aaf5fb4d17e7f050bbf21"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.446765 25241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:43.459676 25241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:43.462579 25241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:43.462762 25241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:43.469873 25241 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.189:33603
I20260620 19:04:43.470283 25308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.189:33603 every 8 connection(s)
I20260620 19:04:43.470883 25241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
I20260620 19:04:43.472256 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25241
I20260620 19:04:43.472393 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-1/wal/instance
I20260620 19:04:43.472677 2882 external_mini_cluster.cc:1428] Setting key 83d77f71bf0a7d82445d9b53e141713e
I20260620 19:04:43.473827 25309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:43.477442 25309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.188@KRBTEST.COM" created.
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:43.491020 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.479314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.2.208.190:46645)
0620 19:04:43.479541 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.479544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.480482 (+ 938us) server_negotiation.cc:407] Connection header received
0620 19:04:43.481410 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.481414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.481474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.481582 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.482704 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.483227 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.484820 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.485045 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.485133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.487443 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.487446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.487476 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.487500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.488979 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.489509 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.489514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.489516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.489597 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.489934 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.489939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.489942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.490125 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.490234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.490728 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.490845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 19:04:43.491954 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.479818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46645 (local address 127.0.0.1:40262)
0620 19:04:43.480343 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.480383 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.481207 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.481747 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.481753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.482071 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.482565 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.482577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.483405 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.483428 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.484684 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.484696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.485194 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.485197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.485371 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.485944 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.485960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.487243 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.489112 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.489118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.489132 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.489401 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.489692 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.489696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.489698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.489837 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.490230 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.490235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.490482 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.490661 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.490970 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":125,"threads_started":1}
I20260620 19:04:43.493391 25309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
I20260620 19:04:43.496104 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:33737
--webserver_interface=127.2.208.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:04:43.502789 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.493863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41226)
0620 19:04:43.493987 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.494002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.494079 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.494586 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.494588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.494784 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.494896 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.494902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.495920 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.495925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.496780 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.496794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.497440 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.497443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.497510 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.497951 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.497961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.499107 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.501313 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.501316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.501318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.501506 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.501827 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.501829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.501830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.501898 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.502452 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.502454 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.502522 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.502596 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.502684 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:04:43.503118 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.493948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.2.208.189:33603)
0620 19:04:43.494268 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.494276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.494293 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:04:43.494326 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.494331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.494472 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.494575 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.494994 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.495796 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.496893 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.497320 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.497436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.499237 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.499249 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.499265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.499295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.501205 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.501616 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.501625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.501627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.501727 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.501986 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.501993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.501996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.502340 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.502487 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.502714 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.502884 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":153,"threads_started":1}
I20260620 19:04:43.504895 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } attempt: 3
I20260620 19:04:43.505169 25309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:43.506103 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.505893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40394)
0620 19:04:43.505977 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.506050 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:04:43.506357 25257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
I20260620 19:04:43.507354 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.507081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40398)
0620 19:04:43.507214 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.507272 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 19:04:43.508553 25309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:04:43.513959 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.505388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.2.208.189:33603)
0620 19:04:43.505499 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.505503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.505593 (+ 90us) server_negotiation.cc:407] Connection header received
0620 19:04:43.505733 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.505737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.505785 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.505860 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.506395 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.507142 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.508466 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.508810 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.508917 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.511003 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.511007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.511010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.511032 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.512417 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.512845 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.512849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.512851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.512928 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.513193 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.513197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.513199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.513355 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.513456 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.513700 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.513814 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:04:43.513962 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.505350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41236)
0620 19:04:43.505506 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.505522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.505640 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.505894 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.505899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.506112 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.506272 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.506280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.507251 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.507255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.508351 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.508360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.508919 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.508930 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.509035 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.509532 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.509546 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.510855 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.512544 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.512547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.512550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.512750 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.513024 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.513027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.513029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.513108 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.513454 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.513457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.513551 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.513654 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.513801 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 19:04:43.515028 25234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:43.515739 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.515530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40410)
0620 19:04:43.515632 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.515687 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:04:43.516377 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.516153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40414)
0620 19:04:43.516279 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.516322 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 19:04:43.516515 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
I20260620 19:04:43.547660 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 1
I20260620 19:04:43.548555 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.548245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40422)
0620 19:04:43.548408 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.548473 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 19:04:43.549314 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.549076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40434)
0620 19:04:43.549180 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.549250 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:04:43.549451 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
I20260620 19:04:43.557809 25309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 1
I20260620 19:04:43.558509 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.558230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40450)
0620 19:04:43.558372 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.558432 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 19:04:43.559222 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.558973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40462)
0620 19:04:43.559094 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.559162 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 19:04:43.559422 25309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
W20260620 19:04:43.609628 25315 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:43.609885 25315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:43.609910 25315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:43.610023 25315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:43.614137 25315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:43.614188 25315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:43.614208 25315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:43.614264 25315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:43.614297 25315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:43.619253 25315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:46645,127.2.208.189:33603,127.2.208.188:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:33737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.208.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:43.620222 25315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:43.621225 25315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:04:43.624599 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 2
I20260620 19:04:43.625373 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40466)
0620 19:04:43.625216 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.625280 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:04:43.626108 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.625870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40480)
0620 19:04:43.625985 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.626045 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 19:04:43.626255 25234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
I20260620 19:04:43.627576 25309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 2
I20260620 19:04:43.628219 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.627963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40492)
0620 19:04:43.628094 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.628150 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:04:43.628976 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.628737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40508)
0620 19:04:43.628851 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.628908 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:04:43.629127 25309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:33737: Network error: Client connection negotiation failed: client connection to 127.2.208.188:33737: connect: Connection refused (error 111)
W20260620 19:04:43.630179 25321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:43.630162 25324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:43.630189 25322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:43.630659 25315 server_base.cc:1034] running on GCE node
I20260620 19:04:43.631193 25315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:43.632254 25315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:43.633466 25315 hybrid_clock.cc:648] HybridClock initialized: now 1781982283633455 us; error 31 us; skew 500 ppm
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:43.636119 25315 init.cc:377] Logged in from keytab as kudu/127.2.208.188@KRBTEST.COM (short username kudu)
I20260620 19:04:43.636898 25315 webserver.cc:460] Webserver started at http://127.2.208.188:39209/ using document root <none> and password file <none>
I20260620 19:04:43.637410 25315 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:43.637494 25315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:43.637707 25315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:43.639739 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data/instance:
uuid: "aa99e0fe1cb0498bb9f6609df607c3b4"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "f7126c4d4a795b702f6dd8310f4a8a68"
server_key_iv: "ab4bd3affa9c1f9fa6729d37a95bd4c0"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.640283 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal/instance:
uuid: "aa99e0fe1cb0498bb9f6609df607c3b4"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "f7126c4d4a795b702f6dd8310f4a8a68"
server_key_iv: "ab4bd3affa9c1f9fa6729d37a95bd4c0"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.643718 25315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:43.645994 25331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:43.647255 25315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:43.647434 25315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "aa99e0fe1cb0498bb9f6609df607c3b4"
format_stamp: "Formatted at 2026-06-20 19:04:43 on dist-test-slave-tsm4"
server_key: "f7126c4d4a795b702f6dd8310f4a8a68"
server_key_iv: "ab4bd3affa9c1f9fa6729d37a95bd4c0"
server_key_version: "encryptionkey@0"
I20260620 19:04:43.647567 25315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:43.669569 25315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:43.672453 25315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:43.672648 25315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:43.679222 25315 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.188:33737
I20260620 19:04:43.679508 25383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.188:33737 every 8 connection(s)
I20260620 19:04:43.680226 25315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
I20260620 19:04:43.682170 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25315
I20260620 19:04:43.682302 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982008625501-2882-0/minicluster-data/master-2/wal/instance
I20260620 19:04:43.682624 2882 external_mini_cluster.cc:1428] Setting key dd3846676053715a0547f21b2560a042
I20260620 19:04:43.683243 25384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:43.687642 25384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:43.697353 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.683867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.2.208.188:33737)
0620 19:04:43.684238 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.684252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.684286 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:04:43.684940 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.684956 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.685262 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.685588 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.686372 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.687435 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.688357 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.688822 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.688989 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.691776 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.691782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.691793 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.691830 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.694376 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.694901 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.694906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.694911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.695014 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.695327 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.695332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.695334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.695795 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.696001 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.696267 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.696546 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260620 19:04:43.698756 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.689237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:46645 (local address 127.0.0.1:40274)
0620 19:04:43.689787 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.689834 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.689968 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.690288 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.690292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.690596 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.690779 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.690789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.691610 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.691614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.692614 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.692624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.693085 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.693089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.693205 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.693707 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.693726 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.695437 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.697135 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.697143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.697147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.697422 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.697776 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.697778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.697780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.697864 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.698245 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.698250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.698361 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.698505 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.698630 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"spinlock_wait_cycles":5376,"thread_start_us":131,"threads_started":1}
I20260620 19:04:43.698863 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.689320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.2.208.190:46645)
0620 19:04:43.689453 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.689456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.689937 (+ 481us) server_negotiation.cc:407] Connection header received
0620 19:04:43.690076 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.690082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.690144 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.690245 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.690914 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.691494 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.692753 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.692978 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.693084 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.695561 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.695563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.695565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.695587 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.697030 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.697566 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.697569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.697571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.697662 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.697960 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.697963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.697964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.698142 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.698219 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.698589 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.698704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:04:43.699816 25384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:43.707803 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.700271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:33603 (local address 127.0.0.1:41250)
0620 19:04:43.700444 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.700457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.700525 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.700740 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.700745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.700877 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.700967 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.700973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.701840 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.701843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.702525 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.702533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.703004 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.703008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.703119 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.703553 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.703566 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.704669 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.706472 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.706475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.706476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.706640 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.706888 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.706890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.706891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.706955 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.707292 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.707295 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.707423 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.707538 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.707655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 19:04:43.707824 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.700388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.2.208.189:33603)
0620 19:04:43.700514 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.700518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.700538 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:04:43.700600 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.700602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.700637 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.700698 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.701093 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.701741 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.702628 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.702882 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.702974 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.704796 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.704799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.704801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.704820 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.706366 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.706732 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.706737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.706740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.706813 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.707026 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.707028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.707029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.707188 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.707296 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.707597 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.707696 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 19:04:43.708562 25384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:43.708761 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.699698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.2.208.190:46645)
0620 19:04:43.699793 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.699797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.699830 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:04:43.699913 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.699916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.699954 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.700032 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.700759 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.701420 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.702346 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.702548 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.702658 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.705480 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.705485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.705489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.705518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.706802 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.707313 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.707318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.707320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.707422 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.707692 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.707696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.707697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.708076 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.708265 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.708457 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.708592 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:04:43.717077 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.709253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40512)
0620 19:04:43.709368 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.709378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.709461 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.709986 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.709990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.710152 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.710234 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.710241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.710902 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.710905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.711465 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.711480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.711891 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.711894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.712020 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.712432 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.712445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.713618 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.715418 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.715421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.715423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.715607 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.715891 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.715893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.715894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.715963 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.716668 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.716671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.716759 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.716835 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.716919 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":42}
I20260620 19:04:43.717228 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.709391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.2.208.188:33737)
0620 19:04:43.709687 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.709690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.709700 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:04:43.709724 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.709726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.709754 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.709840 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.710317 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.710796 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.711574 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.711792 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.711894 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.713744 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.713747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.713749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.713768 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.715280 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.715708 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.715712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.715714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.715797 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.716362 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.716365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.716368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.716576 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.716665 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.716908 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.717021 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260620 19:04:43.717707 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.709884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.2.208.189:33603)
0620 19:04:43.709980 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.709984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.709996 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:04:43.710066 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.710068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.710109 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.710180 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.710708 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.711174 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.711971 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.712192 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.712306 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.714555 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.714560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.714563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.714589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.715961 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.716334 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.716338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.716340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.716421 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.716696 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.716698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.716700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.716892 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.717111 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.717527 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.717612 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:04:43.720329 25309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 3
I20260620 19:04:43.720800 25384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: Bootstrap starting.
I20260620 19:04:43.720929 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:04:43.722191 25208 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc728e22dd7f43fdb709744473a46887"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40278
I20260620 19:04:43.723479 25384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:43.724371 25384 log.cc:824] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:04:43.725394 25234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } attempt: 3
I20260620 19:04:43.726565 25384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: No bootstrap required, opened a new log
I20260620 19:04:43.729075 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.720678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40516)
0620 19:04:43.720839 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.720852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.720932 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.721237 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.721239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.721400 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.721494 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.721500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.722403 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.722406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.723107 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.723118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.723717 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.723722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.723805 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.724319 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.724332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.725568 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.727611 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.727614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.727617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.727800 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.728078 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.728081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.728083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.728178 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.728546 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.728548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.728674 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.728778 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.728905 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 19:04:43.729274 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.720754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.2.208.188:33737)
0620 19:04:43.720845 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.720849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.721037 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:04:43.721071 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.721074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.721110 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.721193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.721615 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.722290 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.723235 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.723586 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.723680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.725711 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.725714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.725716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.725737 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.727486 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.727908 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.727913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.727915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.727998 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.728269 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.728272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.728274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.728469 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.728549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.728889 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.729039 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:04:43.730757 25384 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.731060 25384 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa99e0fe1cb0498bb9f6609df607c3b4, State: Initialized, Role: FOLLOWER
Jun 20 19:04:43 dist-test-slave-tsm4 krb5kdc[25153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982283, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:04:43.731444 25384 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.732064 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.732254 25309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: Bootstrap starting.
I20260620 19:04:43.732393 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.732971 25384 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:43.734949 25309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:43.735745 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.725780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:33737 (local address 127.0.0.1:40526)
0620 19:04:43.725941 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:43.725954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:43.726027 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:43.726769 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:43.726773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:43.726953 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:43.727047 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.727054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.728026 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.728029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.728833 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:43.728843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.729876 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.729880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:43.729975 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.730431 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:43.730445 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:43.731865 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:43.734084 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.734087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.734089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.734354 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.734694 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:43.734698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:43.734702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:43.734805 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:43.735254 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:43.735257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:43.735358 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 19:04:43.735474 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 19:04:43.735599 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 19:04:43.735908 25309 log.cc:824] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:43.736331 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.726208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.2.208.188:33737)
0620 19:04:43.726500 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.726511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.726527 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:04:43.726558 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.726560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.726615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.726745 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.727172 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.727899 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.728972 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.729743 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.730171 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.732035 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.732039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.732041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.732063 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.733956 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.734485 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.734489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.734491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.734577 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.734937 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.734940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.734942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.735145 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.736006 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.736076 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.736176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260620 19:04:43.738593 25309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: No bootstrap required, opened a new log
I20260620 19:04:43.739553 25234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: Bootstrap starting.
I20260620 19:04:43.736720 25315 master_runner.cc:186] Error getting master registration for 127.2.208.190:46645: OK, instance_id { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" instance_seqno: 1781982283264245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:04:43.739964 25315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:04:43.741858 25309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.742026 25309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f392942a42c2b66cd04e67c6a0ec, State: Initialized, Role: FOLLOWER
I20260620 19:04:43.742101 25234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:43.742268 25309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.742623 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.742735 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.742936 25234 log.cc:824] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:43.743345 25309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:04:43.744817 25402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:43.744890 25402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:43.745045 25234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: No bootstrap required, opened a new log
I20260620 19:04:43.748134 25234 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.748334 25234 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc728e22dd7f43fdb709744473a46887, State: Initialized, Role: FOLLOWER
W20260620 19:04:43.748466 25417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:43.748538 25417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:43.748647 25234 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.749112 25419 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.749329 25419 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.746651 25241 master_runner.cc:186] Error getting master registration for 127.2.208.190:46645: OK, instance_id { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" instance_seqno: 1781982283264245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:04:43.749846 25234 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:43.749931 25208 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:04:43.749979 25241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:04:43.750018 25208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:43.751385 25208 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.752844 25283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc728e22dd7f43fdb709744473a46887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec"
I20260620 19:04:43.752835 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc728e22dd7f43fdb709744473a46887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4"
I20260620 19:04:43.753062 25358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:43.753052 25283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:43.754315 25283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc728e22dd7f43fdb709744473a46887 in term 1.
I20260620 19:04:43.752341 25208 leader_election.cc:288] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [CANDIDATE]: Term 1 election: Requested vote from peers 5aa9f392942a42c2b66cd04e67c6a0ec (127.2.208.189:33603), aa99e0fe1cb0498bb9f6609df607c3b4 (127.2.208.188:33737)
I20260620 19:04:43.754747 25358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc728e22dd7f43fdb709744473a46887 in term 1.
I20260620 19:04:43.754733 25184 leader_election.cc:302] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f392942a42c2b66cd04e67c6a0ec, bc728e22dd7f43fdb709744473a46887; no voters:
I20260620 19:04:43.755558 25419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:43.755752 25419 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [term 1 LEADER]: Becoming Leader. State: Replica: bc728e22dd7f43fdb709744473a46887, State: Running, Role: LEADER
I20260620 19:04:43.756083 25419 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } }
I20260620 19:04:43.757730 25419 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc728e22dd7f43fdb709744473a46887. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.757822 25419 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: This master's current role is: LEADER
W20260620 19:04:43.760742 25187 tablet.cc:2369] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:43.761122 25166 master_runner.cc:263] No leader master found from master bc728e22dd7f43fdb709744473a46887
I20260620 19:04:43.761199 25166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:04:43.768467 25430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:43.768533 25430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:43.770219 25437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:43.771489 25437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:43.773762 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.763726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.2.208.188:33737)
0620 19:04:43.763849 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.763853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.764175 (+ 322us) server_negotiation.cc:407] Connection header received
0620 19:04:43.764362 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.764365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.764413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.764462 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.765500 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.766276 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.767522 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.767839 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.767926 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.769957 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.769961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.769963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.769988 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.771829 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.772390 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.772394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.772396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.772482 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.772775 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.772778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.772780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.773004 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.773142 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.773503 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.773654 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:04:43.782960 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.764227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.2.208.189:33603)
0620 19:04:43.764341 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.764345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.772305 (+ 7960us) server_negotiation.cc:407] Connection header received
0620 19:04:43.773081 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.773085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.773137 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.773192 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.775498 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.776228 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.777295 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.777624 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.777692 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.779309 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.779313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.779316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.779340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.781147 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.781683 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.781687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.781688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.781769 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.782065 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.782068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.782070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.782279 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.782370 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.782687 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.782798 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:04:43.782961 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:43.763791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.2.208.190:46645)
0620 19:04:43.763942 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.763946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.771625 (+ 7679us) server_negotiation.cc:407] Connection header received
0620 19:04:43.771793 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.771796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.771840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.771889 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.774033 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.774754 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.776870 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:43.777215 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:43.777286 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:43.779035 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:43.779039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:43.779042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:43.779067 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:43.781042 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.781595 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.781599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.781601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.781685 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:43.781994 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:43.781998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:43.782000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:43.782222 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:43.782322 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:43.782675 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 19:04:43.782790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 19:04:43.791368 25358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [term 1 FOLLOWER]: Refusing update from remote peer bc728e22dd7f43fdb709744473a46887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:43.791563 25283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [term 1 FOLLOWER]: Refusing update from remote peer bc728e22dd7f43fdb709744473a46887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:43.792322 25431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:43.793334 25431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:43.796922 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc728e22dd7f43fdb709744473a46887. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.797040 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.797142 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc728e22dd7f43fdb709744473a46887. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.797242 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.799647 25431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.799770 25431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:43.801914 25419 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.802040 25419 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887 [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:43.803033 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.803139 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.805109 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc728e22dd7f43fdb709744473a46887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc728e22dd7f43fdb709744473a46887" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 46645 } } peers { permanent_uuid: "5aa9f392942a42c2b66cd04e67c6a0ec" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 33603 } } peers { permanent_uuid: "aa99e0fe1cb0498bb9f6609df607c3b4" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 33737 } } }
I20260620 19:04:43.805218 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:43.805626 25442 mvcc.cc:205] Tried to move back new op lower bound from 7298999434402820096 to 7298999434268004352. Current Snapshot: MvccSnapshot[applied={T|T < 7298999434402820096}]
I20260620 19:04:43.807533 25437 catalog_manager.cc:1347] Generated new cluster ID: 4aafd5a516154ffc89fc93225e5b1c66
I20260620 19:04:43.807605 25437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:43.812872 25444 mvcc.cc:205] Tried to move back new op lower bound from 7298999434402820096 to 7298999434268004352. Current Snapshot: MvccSnapshot[applied={T|T < 7298999434402820096}]
I20260620 19:04:43.838526 25437 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:43.839803 25437 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:43.853376 25437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc728e22dd7f43fdb709744473a46887: Generated new TSK 0
I20260620 19:04:43.854276 25437 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:04:43.878393 25390 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.875352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.2.208.188:33737)
0620 19:04:43.875503 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.875508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.877234 (+ 1726us) server_negotiation.cc:407] Connection header received
0620 19:04:43.877496 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.877500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.877551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.877604 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.878238 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40548: BlockingRecv error: recv got EOF from 127.0.0.1:40548 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 19:04:43.879765 25238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.875883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.2.208.190:46645)
0620 19:04:43.876008 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.876012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.878869 (+ 2857us) server_negotiation.cc:407] Connection header received
0620 19:04:43.879036 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.879040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.879087 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.879136 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.879647 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40302: BlockingRecv error: recv got EOF from 127.0.0.1:40302 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 19:04:43.881104 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.876644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.2.208.189:33603)
0620 19:04:43.876755 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.876759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.880210 (+ 3451us) server_negotiation.cc:407] Connection header received
0620 19:04:43.880398 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.880401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.880450 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.880501 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.880986 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41278: BlockingRecv error: recv got EOF from 127.0.0.1:41278 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:04:43.885542 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.884045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.2.208.189:33603)
0620 19:04:43.884152 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.884156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.884615 (+ 459us) server_negotiation.cc:407] Connection header received
0620 19:04:43.884823 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.884826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.884901 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.884951 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.885440 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41284: BlockingRecv error: recv got EOF from 127.0.0.1:41284 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 19:04:43.890636 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.889121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.2.208.189:33603)
0620 19:04:43.889221 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.889224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.889772 (+ 548us) server_negotiation.cc:407] Connection header received
0620 19:04:43.889953 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.889956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.890006 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.890053 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.890538 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41294: BlockingRecv error: recv got EOF from 127.0.0.1:41294 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 19:04:43.896296 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.894751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.2.208.189:33603)
0620 19:04:43.894852 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.894856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.895352 (+ 496us) server_negotiation.cc:407] Connection header received
0620 19:04:43.895626 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.895629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.895677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.895725 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.896201 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41300: BlockingRecv error: recv got EOF from 127.0.0.1:41300 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 19:04:43.904577 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.902748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.2.208.189:33603)
0620 19:04:43.902932 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.902937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.903543 (+ 606us) server_negotiation.cc:407] Connection header received
0620 19:04:43.903756 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.903760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.903818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.903873 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.904456 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41304: BlockingRecv error: recv got EOF from 127.0.0.1:41304 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 19:04:43.917012 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.915162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.2.208.189:33603)
0620 19:04:43.915289 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.915293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.915957 (+ 664us) server_negotiation.cc:407] Connection header received
0620 19:04:43.916167 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.916170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.916229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.916285 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.916893 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41312: BlockingRecv error: recv got EOF from 127.0.0.1:41312 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:04:43.937095 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.935508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.2.208.189:33603)
0620 19:04:43.935617 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.935622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.936175 (+ 553us) server_negotiation.cc:407] Connection header received
0620 19:04:43.936362 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.936365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.936415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.936466 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.936983 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41328: BlockingRecv error: recv got EOF from 127.0.0.1:41328 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 19:04:43.972978 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:43.971603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.2.208.189:33603)
0620 19:04:43.971748 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:43.971754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:43.972010 (+ 256us) server_negotiation.cc:407] Connection header received
0620 19:04:43.972194 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:43.972197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:43.972256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:43.972339 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:43.972854 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41344: BlockingRecv error: recv got EOF from 127.0.0.1:41344 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:04:44.040027 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:44.038894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.2.208.189:33603)
0620 19:04:44.039094 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.039098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.039167 (+ 69us) server_negotiation.cc:407] Connection header received
0620 19:04:44.039288 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.039291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.039338 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.039452 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:44.039893 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41356: BlockingRecv error: recv got EOF from 127.0.0.1:41356 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260620 19:04:44.171403 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:44.170119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.2.208.189:33603)
0620 19:04:44.170283 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.170287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.170419 (+ 132us) server_negotiation.cc:407] Connection header received
0620 19:04:44.170619 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.170622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.170669 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.170757 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:44.171234 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41370: BlockingRecv error: recv got EOF from 127.0.0.1:41370 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 19:04:44.430751 25314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:44.429598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.2.208.189:33603)
0620 19:04:44.429733 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.429737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.429843 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:04:44.430014 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.430017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.430070 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.430167 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:44.430632 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41378: BlockingRecv error: recv got EOF from 127.0.0.1:41378 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:04:44.746275 25402 catalog_manager.cc:1259] Loaded cluster ID: 4aafd5a516154ffc89fc93225e5b1c66
I20260620 19:04:44.746351 25402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: loading cluster ID for follower catalog manager: success
I20260620 19:04:44.748062 25402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: acquiring CA information for follower catalog manager: success
I20260620 19:04:44.749164 25402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa99e0fe1cb0498bb9f6609df607c3b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:04:44.749596 25417 catalog_manager.cc:1259] Loaded cluster ID: 4aafd5a516154ffc89fc93225e5b1c66
I20260620 19:04:44.749673 25417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: loading cluster ID for follower catalog manager: success
I20260620 19:04:44.751456 25417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: acquiring CA information for follower catalog manager: success
I20260620 19:04:44.752501 25417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5aa9f392942a42c2b66cd04e67c6a0ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:04:44.948618 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:44.944933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.2.208.189:33603)
0620 19:04:44.945183 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.945187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.945287 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:04:44.945395 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.945397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.945518 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.945620 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:44.945864 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.946370 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.947219 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.947453 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.947532 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:44.947839 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:44.948293 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:44.948386 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:44.948451 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:04:44.948496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260620 19:04:44.955547 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:44.951866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.2.208.188:33737)
0620 19:04:44.952127 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.952133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.952147 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:44.952258 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.952264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.952434 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.952562 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:44.952765 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.953275 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.954065 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.954304 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.954412 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:44.954709 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:44.955121 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:44.955265 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:44.955299 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:04:44.955350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":56,"threads_started":1}
I20260620 19:04:44.962271 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:44.958293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.2.208.188:33737)
0620 19:04:44.958390 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.958394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.958468 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:04:44.958601 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.958604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.958651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.958725 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:44.959230 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.960025 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.961243 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.961486 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.961594 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:44.961856 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:44.961927 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:44.962018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:44.962098 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:04:44.962147 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:04:44.962474 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:44.958844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.2.208.189:33603)
0620 19:04:44.959010 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.959013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.959204 (+ 191us) server_negotiation.cc:407] Connection header received
0620 19:04:44.959313 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.959316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.959420 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.959502 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:44.959756 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.960311 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.961296 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.961635 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.961727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:44.961971 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:44.962052 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:44.962118 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:44.962259 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:04:44.962324 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:04:44.963315 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:44.958267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.2.208.190:46645)
0620 19:04:44.958542 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:44.958549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:44.958771 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:04:44.959026 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:44.959031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:44.959212 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:44.959316 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:44.959713 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.960502 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.961635 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:44.961956 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:44.962049 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:44.962461 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:44.962772 (+ 311us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:44.962868 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:44.963152 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 19:04:44.963211 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":128,"threads_started":1}
I20260620 19:04:44.969221 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25166
I20260620 19:04:44.975845 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25241
I20260620 19:04:44.981559 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25315
2026-06-20T19:04:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3034 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:04:45 dist-test-slave-tsm4 krb5kdc[25531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:04:45 dist-test-slave-tsm4 krb5kdc[25531](info): set up 2 sockets
Jun 20 19:04:45 dist-test-slave-tsm4 krb5kdc[25531](info): commencing operation
krb5kdc: starting...
W20260620 19:04:46.015758 2882 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/test-user.keytab.
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.208.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
Entry for principal kudu/127.2.208.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab.
I20260620 19:04:46.132258 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43901
--webserver_interface=127.2.208.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:04:46.238896 25544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:46.239154 25544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:46.239177 25544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:46.239269 25544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:46.243103 25544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:46.243155 25544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:46.243173 25544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:46.243239 25544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:46.243265 25544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:46.248296 25544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.190.keytab
--principal=kudu/127.2.208.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.190:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:46.249289 25544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:46.250332 25544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:46.258574 25552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:46.258574 25549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:46.258646 25550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:46.258726 25544 server_base.cc:1034] running on GCE node
I20260620 19:04:46.259546 25544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:46.260596 25544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:46.261762 25544 hybrid_clock.cc:648] HybridClock initialized: now 1781982286261742 us; error 30 us; skew 500 ppm
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:46.264518 25544 init.cc:377] Logged in from keytab as kudu/127.2.208.190@KRBTEST.COM (short username kudu)
I20260620 19:04:46.265358 25544 webserver.cc:460] Webserver started at http://127.2.208.190:41889/ using document root <none> and password file <none>
I20260620 19:04:46.265946 25544 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:46.266017 25544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:46.266232 25544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:46.268307 25544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data/instance:
uuid: "8b03d5a13f91457da907450c3e65dcca"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "72557b0c7a3a2b321618b18b8d0771b1"
server_key_iv: "ca9e685ac6c914c474459cb545ae83f9"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.269150 25544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance:
uuid: "8b03d5a13f91457da907450c3e65dcca"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "72557b0c7a3a2b321618b18b8d0771b1"
server_key_iv: "ca9e685ac6c914c474459cb545ae83f9"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.272802 25544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:46.275067 25559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:46.276515 25544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:04:46.276611 25544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal
uuid: "8b03d5a13f91457da907450c3e65dcca"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "72557b0c7a3a2b321618b18b8d0771b1"
server_key_iv: "ca9e685ac6c914c474459cb545ae83f9"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.276690 25544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:46.293668 25544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:46.296511 25544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:46.296665 25544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:46.303247 25544 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.190:43901
I20260620 19:04:46.303629 25611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.190:43901 every 8 connection(s)
I20260620 19:04:46.304297 25544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/data/info.pb
I20260620 19:04:46.306951 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:46.307889 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25544
I20260620 19:04:46.308041 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-0/wal/instance
I20260620 19:04:46.308329 2882 external_mini_cluster.cc:1428] Setting key 587f5126501001183c329ba1a72d5b9b
I20260620 19:04:46.310742 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.189@KRBTEST.COM" created.
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:46.326460 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.313024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43901 (local address 127.0.0.1:41300)
0620 19:04:46.313586 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.313621 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.314535 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.315241 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.315246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.315700 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.315958 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.315971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.317318 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.317321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.318051 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.318060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.318857 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.318861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.318986 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.319687 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.319710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.321486 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.323435 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.323447 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.323460 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.323801 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.324179 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.324184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.324189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.324345 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.324956 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.324962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.325145 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.325341 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.325768 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":90,"spinlock_wait_cycles":8448,"thread_start_us":159,"threads_started":1}
I20260620 19:04:46.326467 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.313024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.2.208.190:43901)
0620 19:04:46.313439 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.313462 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.313727 (+ 265us) server_negotiation.cc:407] Connection header received
0620 19:04:46.314856 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.314860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.315045 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.315245 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.316156 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.317134 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.318172 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.318667 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.318841 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.321634 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.321639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.321649 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.321669 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.323298 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.323960 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.323965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.323967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.324074 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.324450 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.324452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.324453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.324827 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.324929 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.325413 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.325674 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
I20260620 19:04:46.329041 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:46.329749 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.329510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52536)
0620 19:04:46.329624 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.329696 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:04:46.329941 25563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
I20260620 19:04:46.330760 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.330566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52550)
0620 19:04:46.330669 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.330717 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:04:46.330915 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:42881: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
Entry for principal kudu/127.2.208.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab.
I20260620 19:04:46.334650 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:42881
--webserver_interface=127.2.208.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:04:46.380115 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } attempt: 1
I20260620 19:04:46.381057 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.380717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52560)
0620 19:04:46.380889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.380965 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 19:04:46.381807 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.381581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52572)
0620 19:04:46.381691 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.381745 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:04:46.381989 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:42881: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
W20260620 19:04:46.442883 25619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:46.443137 25619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:46.443159 25619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:46.443249 25619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:46.447184 25619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:46.447234 25619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:46.447253 25619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:46.447268 25619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:46.447283 25619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:46.450191 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } attempt: 2
I20260620 19:04:46.451021 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.450705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52582)
0620 19:04:46.450867 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.450936 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 19:04:46.451792 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.451594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52596)
0620 19:04:46.451689 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.451737 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:04:46.451934 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.189:42881: Network error: Client connection negotiation failed: client connection to 127.2.208.189:42881: connect: Connection refused (error 111)
I20260620 19:04:46.452857 25619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.189.keytab
--principal=kudu/127.2.208.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.189:42881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.208.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:46.453840 25619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:46.454825 25619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:04:46.463819 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:46.463835 25627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:46.463819 25625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:46.464082 25619 server_base.cc:1034] running on GCE node
I20260620 19:04:46.464700 25619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:46.465723 25619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:46.466905 25619 hybrid_clock.cc:648] HybridClock initialized: now 1781982286466892 us; error 29 us; skew 500 ppm
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:46.469969 25619 init.cc:377] Logged in from keytab as kudu/127.2.208.189@KRBTEST.COM (short username kudu)
I20260620 19:04:46.470824 25619 webserver.cc:460] Webserver started at http://127.2.208.189:33043/ using document root <none> and password file <none>
I20260620 19:04:46.471444 25619 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:46.471534 25619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:46.471735 25619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:46.473778 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data/instance:
uuid: "933a0152e96c44398ad9b796887d5a61"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "8b86da966ff8aface567caae5627c059"
server_key_iv: "9b2e1d3d58915b587e96a4aa2f293761"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.474349 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal/instance:
uuid: "933a0152e96c44398ad9b796887d5a61"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "8b86da966ff8aface567caae5627c059"
server_key_iv: "9b2e1d3d58915b587e96a4aa2f293761"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.478132 25619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:04:46.480633 25634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:46.481861 25619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:04:46.482009 25619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal
uuid: "933a0152e96c44398ad9b796887d5a61"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "8b86da966ff8aface567caae5627c059"
server_key_iv: "9b2e1d3d58915b587e96a4aa2f293761"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.482116 25619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:46.496625 25619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:46.499199 25619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:46.499454 25619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:46.506439 25619 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.189:42881
I20260620 19:04:46.506708 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.189:42881 every 8 connection(s)
I20260620 19:04:46.507337 25619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/data/info.pb
I20260620 19:04:46.510120 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:46.513005 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25619
I20260620 19:04:46.513145 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-1/wal/instance
I20260620 19:04:46.513422 2882 external_mini_cluster.cc:1428] Setting key a1acf0bc45d28586cf4de0847c0dea73
I20260620 19:04:46.513840 25687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.208.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.208.188@KRBTEST.COM" created.
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:46.528450 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.515730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.2.208.190:43901)
0620 19:04:46.515874 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.515878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.516826 (+ 948us) server_negotiation.cc:407] Connection header received
0620 19:04:46.517801 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.517808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.517890 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.518002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.519099 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.519979 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.521101 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.521526 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.521613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.524689 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.524692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.524694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.524718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.526410 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.526924 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.526929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.526930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.527015 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.527343 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.527348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.527350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.527561 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.527668 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.528154 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.528278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:04:46.529208 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.516204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43901 (local address 127.0.0.1:41310)
0620 19:04:46.516651 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.516701 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.517587 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.518152 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.518158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.518508 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.518947 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.518959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.520106 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.520108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.520976 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.520985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.521659 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.521661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.521797 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.522332 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.522348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.524509 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.526535 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.526541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.526555 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.526817 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.527107 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.527111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.527113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.527246 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.527666 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.527673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.527912 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.528106 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.528413 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":126,"threads_started":1}
I20260620 19:04:46.530673 25687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
Entry for principal kudu/127.2.208.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab.
I20260620 19:04:46.538371 2882 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
/tmp/dist-test-taskDDC2sW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:35473
--webserver_interface=127.2.208.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.208.148:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:04:46.539973 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.531161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52600)
0620 19:04:46.531277 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.531291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.531391 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.532015 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.532017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.532143 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.532230 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.532234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.533231 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.533233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.533921 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.533954 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.534528 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.534531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.534646 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.535119 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.535135 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.536456 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.538331 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.538337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.538340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.538568 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.538868 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.538874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.538879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.538993 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.539470 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.539477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.539589 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.539714 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.539857 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:04:46.540345 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.531330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.2.208.189:42881)
0620 19:04:46.531671 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.531680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.531698 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:04:46.531731 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.531735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.531873 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.532033 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.532329 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.533102 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.534071 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.534426 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.534510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.536578 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.536586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.536596 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.536615 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.538223 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.538689 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.538694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.538696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.538771 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.539087 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.539090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.539091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.539304 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.539513 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.539829 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.539988 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":176,"threads_started":1}
I20260620 19:04:46.542299 25687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:46.543023 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.542783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48760)
0620 19:04:46.542898 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.542973 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:04:46.543134 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } attempt: 3
W20260620 19:04:46.543202 25637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
I20260620 19:04:46.544201 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.543908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48768)
0620 19:04:46.544058 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.544122 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 19:04:46.544490 25687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:04:46.553153 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.544069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.2.208.189:42881)
0620 19:04:46.544169 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.544173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.544184 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:04:46.544218 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.544220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.544277 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.544355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.544902 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.545386 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.547198 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.547492 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.547568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.549687 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.549691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.549694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.549723 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.551621 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.552027 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.552034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.552038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.552112 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.552396 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.552400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.552402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.552562 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.552666 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.552913 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.553025 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:04:46.553154 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.543602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52612)
0620 19:04:46.543798 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.543813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.543897 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.544394 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.544400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.544626 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.544779 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.544789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.545510 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.545514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.547075 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.547091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.547605 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.547612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.547755 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.548196 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.548217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.549560 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.551729 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.551733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.551735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.551932 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.552202 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.552206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.552208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.552308 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.552657 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.552660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.552760 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.552865 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.552994 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 19:04:46.553889 25612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:46.554545 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.554328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48784)
0620 19:04:46.554449 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.554493 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 19:04:46.555179 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.554956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48800)
0620 19:04:46.555084 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.555127 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 19:04:46.555320 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
I20260620 19:04:46.586511 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 1
I20260620 19:04:46.587440 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.587066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48802)
0620 19:04:46.587243 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.587320 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 19:04:46.588203 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.587970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48804)
0620 19:04:46.588083 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.588137 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:04:46.588336 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
I20260620 19:04:46.593758 25687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 1
I20260620 19:04:46.594512 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.594217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48816)
0620 19:04:46.594362 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.594426 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 19:04:46.595204 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.594994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48824)
0620 19:04:46.595094 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.595144 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:04:46.595352 25687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
W20260620 19:04:46.650669 25693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:04:46.650930 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:04:46.650952 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:04:46.651063 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:04:46.655081 25693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:04:46.655156 25693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:04:46.655193 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:04:46.655249 25693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:04:46.655288 25693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:04:46.660319 25693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.208.148:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.208.190:43901,127.2.208.189:42881,127.2.208.188:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/krb5kdc/kudu_127.2.208.188.keytab
--principal=kudu/127.2.208.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.208.188:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.208.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:04:46.661324 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:04:46.662297 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:04:46.663499 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 2
I20260620 19:04:46.663549 25687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 2
I20260620 19:04:46.664326 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.664023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48832)
0620 19:04:46.664167 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.664233 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 19:04:46.664358 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.664052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48840)
0620 19:04:46.664246 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.664293 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 19:04:46.665066 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.664828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48846)
0620 19:04:46.664945 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.665002 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:04:46.665169 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.664884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48854)
0620 19:04:46.665050 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.665098 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 19:04:46.665180 25687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
W20260620 19:04:46.665282 25612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.208.188:35473: Network error: Client connection negotiation failed: client connection to 127.2.208.188:35473: connect: Connection refused (error 111)
W20260620 19:04:46.670951 25702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:04:46.670940 25699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:46.671038 25693 server_base.cc:1034] running on GCE node
W20260620 19:04:46.670951 25700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:04:46.671702 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:04:46.672691 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:04:46.673899 25693 hybrid_clock.cc:648] HybridClock initialized: now 1781982286673860 us; error 24 us; skew 500 ppm
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:04:46.676539 25693 init.cc:377] Logged in from keytab as kudu/127.2.208.188@KRBTEST.COM (short username kudu)
I20260620 19:04:46.677443 25693 webserver.cc:460] Webserver started at http://127.2.208.188:40979/ using document root <none> and password file <none>
I20260620 19:04:46.677977 25693 fs_manager.cc:359] Metadata directory not provided
I20260620 19:04:46.678066 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:04:46.678297 25693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:04:46.680344 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data/instance:
uuid: "64f54b8044ee4143b37a27d526f18bdf"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "20f7a9d68b6cb45e9cd8471c7021aa5a"
server_key_iv: "1ef8a8496e214f790e49963e6ab2e6df"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.680867 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal/instance:
uuid: "64f54b8044ee4143b37a27d526f18bdf"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "20f7a9d68b6cb45e9cd8471c7021aa5a"
server_key_iv: "1ef8a8496e214f790e49963e6ab2e6df"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.684479 25693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:04:46.686833 25709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:04:46.688081 25693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:04:46.688176 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data,/tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal
uuid: "64f54b8044ee4143b37a27d526f18bdf"
format_stamp: "Formatted at 2026-06-20 19:04:46 on dist-test-slave-tsm4"
server_key: "20f7a9d68b6cb45e9cd8471c7021aa5a"
server_key_iv: "1ef8a8496e214f790e49963e6ab2e6df"
server_key_version: "encryptionkey@0"
I20260620 19:04:46.688309 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:04:46.705132 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:04:46.707988 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:04:46.708174 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:04:46.715018 25693 rpc_server.cc:307] RPC server started. Bound to: 127.2.208.188:35473
I20260620 19:04:46.715291 25761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.208.188:35473 every 8 connection(s)
I20260620 19:04:46.715963 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/data/info.pb
I20260620 19:04:46.718627 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:04:46.722134 25762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:46.725001 2882 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu as pid 25693
I20260620 19:04:46.725135 2882 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDDC2sW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982008625501-2882-0/minicluster-data/master-2/wal/instance
I20260620 19:04:46.725450 2882 external_mini_cluster.cc:1428] Setting key 0add83fca1469e74b6f26d365a0b8070
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:04:46.736748 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.723919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.2.208.190:43901)
0620 19:04:46.724075 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.724078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.724854 (+ 776us) server_negotiation.cc:407] Connection header received
0620 19:04:46.725989 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.725992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.726049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.726179 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.727625 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.728460 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.729764 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.730022 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.730118 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.732781 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.732784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.732786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.732810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.734490 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.735044 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.735050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.735052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.735137 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.735595 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.735599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.735600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.735854 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.735959 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.736441 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.736589 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:04:46.737396 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.724271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.190:43901 (local address 127.0.0.1:41324)
0620 19:04:46.724680 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.724720 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.725783 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.726348 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.726357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.726847 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.727490 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.727506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.728589 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.728591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.729648 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.729659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.730158 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.730161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.730298 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.730879 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.730905 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.732621 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.734590 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.734597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.734604 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.734944 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.735235 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.735237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.735239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.735467 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.735975 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.735979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.736196 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.736357 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.736649 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260620 19:04:46.737397 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.726110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.2.208.188:35473)
0620 19:04:46.726418 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.726429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.726567 (+ 138us) server_negotiation.cc:407] Connection header received
0620 19:04:46.726640 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.726645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.726835 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.726995 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.727720 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.728878 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.729948 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.730405 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.730483 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.733310 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.733316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.733328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.733353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.735122 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.735604 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.735610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.735612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.735686 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.735945 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.735948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.735950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.736205 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.736383 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.736608 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.736728 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":226,"spinlock_wait_cycles":1280,"thread_start_us":96,"threads_started":1}
I20260620 19:04:46.738849 25762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.190@KRBTEST.COM
I20260620 19:04:46.747681 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.739248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.189:42881 (local address 127.0.0.1:52618)
0620 19:04:46.739401 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.739413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.739500 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.739867 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.739870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.740059 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.740168 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.740175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.741099 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.741104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.741624 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.741632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.742065 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.742068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.742165 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.742985 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.742998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.744301 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.746231 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.746252 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.746256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.746461 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.746745 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.746748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.746749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.746843 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.747245 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.747248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.747342 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.747469 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.747577 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 19:04:46.747818 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.739416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.2.208.189:42881)
0620 19:04:46.739571 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.739576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.739621 (+ 45us) server_negotiation.cc:407] Connection header received
0620 19:04:46.739710 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.739713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.739759 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.739844 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.740338 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.740955 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.741762 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.741962 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.742046 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.744441 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.744444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.744446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.744467 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.746127 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.746559 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.746565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.746570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.746651 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.746932 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.746934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.746935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.747157 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.747256 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.747525 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.747644 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:04:46.748412 25762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:04:46.748718 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.740157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.2.208.190:43901)
0620 19:04:46.740266 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.740270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.740322 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:04:46.740439 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.740442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.740480 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.740570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.741162 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.741617 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.742557 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.742765 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.742870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.745109 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.745116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.745118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.745146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.746641 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.747082 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.747085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.747086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.747162 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.747442 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.747447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.747450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.747682 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.747897 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.748467 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.748602 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.188@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.208.189@KRBTEST.COM
I20260620 19:04:46.756322 25687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 3
I20260620 19:04:46.757042 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.748896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48876)
0620 19:04:46.748996 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.749008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.749095 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.749708 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.749714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.749877 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.749965 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.749971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.750855 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.750860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.751433 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.751443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.751877 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.751881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.751959 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.752369 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.752380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.753639 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.755349 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.755355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.755358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.755620 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.755885 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.755888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.755890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.756013 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.756554 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.756556 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.756652 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.756758 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.756872 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:04:46.757541 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.749390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.2.208.188:35473)
0620 19:04:46.749505 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.749509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.749521 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:04:46.749553 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.749556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.749594 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.749670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.750075 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.750735 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.751547 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.751777 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.751849 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.753768 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.753772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.753777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.753808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.755206 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.755717 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.755720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.755721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.755791 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.756095 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.756097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.756098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.756246 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.756347 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.757282 (+ 935us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.757402 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:04:46.758725 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.749905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.2.208.189:42881)
0620 19:04:46.750004 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.750008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.750178 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:04:46.750241 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.750244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.750287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.750362 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.750919 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.751351 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.752307 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.752550 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.752637 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.755131 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.755135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.755137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.755166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.756774 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.757234 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.757237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.757240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.757314 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.757838 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.757842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.757846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.758039 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.758384 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.758461 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.758581 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:04:46.760846 25762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: Bootstrap starting.
I20260620 19:04:46.761988 2882 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.189@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:04:46.763659 25762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:46.763197 25586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8b03d5a13f91457da907450c3e65dcca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41340
I20260620 19:04:46.764514 25612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } attempt: 3
I20260620 19:04:46.764606 25762 log.cc:824] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:46.766180 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.756690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48886)
0620 19:04:46.756857 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.756870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.756937 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.757394 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.757396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.757549 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.757663 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.757670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.759015 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.759018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.759813 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.759824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.760346 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.760350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.760434 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.761140 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.761155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.762513 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.764535 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.764538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.764540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.764773 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.765104 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.765108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.765109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.765194 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.765712 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.765716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.765820 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.765896 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.766030 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:04:46.766721 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.756725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.2.208.188:35473)
0620 19:04:46.757175 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.757185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.757216 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:04:46.757247 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.757250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.757285 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.757377 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.757782 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.758860 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.759931 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.760243 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.760341 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.762650 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.762655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.762657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.762680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.764396 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.764894 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.764900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.764902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.764986 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.765355 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.765361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.765362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.765597 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.766360 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.766437 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.766535 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":103,"threads_started":1}
I20260620 19:04:46.767061 25762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: No bootstrap required, opened a new log
I20260620 19:04:46.769503 25687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: Bootstrap starting.
Jun 20 19:04:46 dist-test-slave-tsm4 krb5kdc[25531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.208.190@KRBTEST.COM for kudu/127.2.208.188@KRBTEST.COM
I20260620 19:04:46.771529 25762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.771787 25762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f54b8044ee4143b37a27d526f18bdf, State: Initialized, Role: FOLLOWER
I20260620 19:04:46.772058 25687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:46.772142 25762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.772773 25769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.772894 25687 log.cc:824] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:46.773062 25769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.773574 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.764882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.208.188:35473 (local address 127.0.0.1:48902)
0620 19:04:46.765037 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:04:46.765049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:04:46.765129 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:04:46.765401 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:04:46.765407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:04:46.765584 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:04:46.765703 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.765710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.766722 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.766742 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.767484 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:04:46.767494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.768083 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.768086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:04:46.768169 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.768652 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:04:46.768666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:04:46.770055 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:04:46.772024 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.772027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.772029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.772268 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.772616 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:04:46.772619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:04:46.772621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:04:46.772704 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:04:46.773097 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:04:46.773100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:04:46.773189 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:04:46.773267 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:04:46.773416 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 19:04:46.773587 25762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:46.774164 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.764919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.2.208.188:35473)
0620 19:04:46.765019 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.765023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.765089 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:04:46.765227 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.765232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.765291 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.765373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.765814 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.766616 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.767623 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.767973 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.768071 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.770199 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.770202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.770205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.770227 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.771902 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.772395 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.772401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.772404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.772496 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.772805 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.772808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.772809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.772983 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.773843 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.773927 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.774030 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:04:46.775259 25687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: No bootstrap required, opened a new log
I20260620 19:04:46.777153 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: Bootstrap starting.
W20260620 19:04:46.778884 25781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:46.779008 25781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:46.779175 25687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.779441 25687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933a0152e96c44398ad9b796887d5a61, State: Initialized, Role: FOLLOWER
I20260620 19:04:46.779798 25612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: Neither blocks nor log segments found. Creating new log.
I20260620 19:04:46.779836 25687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.780650 25612 log.cc:824] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:04:46.777269 25693 master_runner.cc:186] Error getting master registration for 127.2.208.190:43901: OK, instance_id { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" instance_seqno: 1781982286301514 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:04:46.781159 25693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:04:46.781196 25687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:46.782629 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: No bootstrap required, opened a new log
I20260620 19:04:46.785540 25612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.785737 25612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b03d5a13f91457da907450c3e65dcca, State: Initialized, Role: FOLLOWER
I20260620 19:04:46.786056 25612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.786522 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.786769 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.787295 25612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:04:46.787360 25586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:04:46.787483 25586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:46.784543 25619 master_runner.cc:186] Error getting master registration for 127.2.208.190:43901: OK, instance_id { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" instance_seqno: 1781982286301514 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:04:46.788457 25619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:04:46.784217 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.788615 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.788789 25586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
I20260620 19:04:46.789749 25586 leader_election.cc:288] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [CANDIDATE]: Term 1 election: Requested vote from peers 933a0152e96c44398ad9b796887d5a61 (127.2.208.189:42881), 64f54b8044ee4143b37a27d526f18bdf (127.2.208.188:35473)
I20260620 19:04:46.790920 25661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b03d5a13f91457da907450c3e65dcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "933a0152e96c44398ad9b796887d5a61"
I20260620 19:04:46.791150 25661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:04:46.791904 25795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:46.791965 25795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:46.792603 25661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b03d5a13f91457da907450c3e65dcca in term 1.
W20260620 19:04:46.792991 25807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:04:46.793077 25807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:04:46.793030 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b03d5a13f91457da907450c3e65dcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "64f54b8044ee4143b37a27d526f18bdf"
I20260620 19:04:46.793246 25736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:04:46.794265 25736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b03d5a13f91457da907450c3e65dcca in term 1.
I20260620 19:04:46.795601 25563 leader_election.cc:302] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b03d5a13f91457da907450c3e65dcca, 933a0152e96c44398ad9b796887d5a61; no voters:
I20260620 19:04:46.796598 25796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:04:46.796864 25796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [term 1 LEADER]: Becoming Leader. State: Replica: 8b03d5a13f91457da907450c3e65dcca, State: Running, Role: LEADER
I20260620 19:04:46.797228 25796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } }
W20260620 19:04:46.798074 25565 tablet.cc:2369] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:04:46.800844 25544 master_runner.cc:263] No leader master found from master 8b03d5a13f91457da907450c3e65dcca
I20260620 19:04:46.800928 25544 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:04:46.801200 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b03d5a13f91457da907450c3e65dcca. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.801295 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:46.807598 25816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:04:46.808502 25816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:04:46.817166 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.803898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.2.208.189:42881)
0620 19:04:46.804013 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.804017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.807716 (+ 3699us) server_negotiation.cc:407] Connection header received
0620 19:04:46.807926 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.807930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.807980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.808034 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.809098 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.809936 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.811131 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.811528 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.811605 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.813625 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.813630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.813633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.813659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.815357 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.815868 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.815871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.815873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.815954 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.816234 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.816237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.816238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.816438 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.816546 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.816873 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.817001 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:04:46.823809 25736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [term 1 FOLLOWER]: Refusing update from remote peer 8b03d5a13f91457da907450c3e65dcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:46.824054 25661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [term 1 FOLLOWER]: Refusing update from remote peer 8b03d5a13f91457da907450c3e65dcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:04:46.824506 25814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:46.824961 25814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:04:46.828189 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.804367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.2.208.190:43901)
0620 19:04:46.804464 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.804469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.810621 (+ 6152us) server_negotiation.cc:407] Connection header received
0620 19:04:46.815594 (+ 4973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.815598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.815645 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.815694 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.816671 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.817446 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.821532 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.821865 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.821936 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.824291 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.824295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.824297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.824323 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.825818 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.826883 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.826886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.826888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.826968 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.827260 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.827263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.827265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.827482 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.827598 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.827914 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.828038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:04:46.828950 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b03d5a13f91457da907450c3e65dcca. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.829089 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.832648 25769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b03d5a13f91457da907450c3e65dcca. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.832772 25769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.832857 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:46.804722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.2.208.188:35473)
0620 19:04:46.804834 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.804840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.816603 (+ 11763us) server_negotiation.cc:407] Connection header received
0620 19:04:46.818668 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.818672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.818722 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.818771 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.819676 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.820408 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.822400 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:46.822737 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:46.822803 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:46.829463 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:04:46.829467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:04:46.829470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:04:46.829497 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:04:46.831081 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.831571 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.831576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.831579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.831658 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:04:46.831933 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:04:46.831936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:04:46.831938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:04:46.832141 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:04:46.832247 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:46.832563 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 19:04:46.832680 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:04:46.836320 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.836436 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:46.836540 25820 mvcc.cc:205] Tried to move back new op lower bound from 7298999446817361920 to 7298999446730690560. Current Snapshot: MvccSnapshot[applied={T|T < 7298999446817361920}]
I20260620 19:04:46.837641 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.837702 25821 mvcc.cc:205] Tried to move back new op lower bound from 7298999446817361920 to 7298999446730690560. Current Snapshot: MvccSnapshot[applied={T|T < 7298999446817361920}]
I20260620 19:04:46.837765 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.840586 25816 catalog_manager.cc:1347] Generated new cluster ID: 2949fe06eaec4f1688381165d98cbe4d
I20260620 19:04:46.840654 25816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:04:46.850746 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.850874 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca [sys.catalog]: This master's current role is: LEADER
I20260620 19:04:46.851795 25816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:04:46.852181 25769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b03d5a13f91457da907450c3e65dcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b03d5a13f91457da907450c3e65dcca" member_type: VOTER last_known_addr { host: "127.2.208.190" port: 43901 } } peers { permanent_uuid: "933a0152e96c44398ad9b796887d5a61" member_type: VOTER last_known_addr { host: "127.2.208.189" port: 42881 } } peers { permanent_uuid: "64f54b8044ee4143b37a27d526f18bdf" member_type: VOTER last_known_addr { host: "127.2.208.188" port: 35473 } } }
I20260620 19:04:46.852293 25769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:04:46.852420 25823 mvcc.cc:205] Tried to move back new op lower bound from 7298999446817361920 to 7298999446730690560. Current Snapshot: MvccSnapshot[applied={T|T < 7298999446817361920}]
I20260620 19:04:46.852902 25816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:04:46.869199 25816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b03d5a13f91457da907450c3e65dcca: Generated new TSK 0
I20260620 19:04:46.869990 25816 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:04:46.905633 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.902580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.2.208.189:42881)
0620 19:04:46.902707 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.902712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.904575 (+ 1863us) server_negotiation.cc:407] Connection header received
0620 19:04:46.904766 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.904769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.904818 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.904870 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.905487 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52642: BlockingRecv error: recv got EOF from 127.0.0.1:52642 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 19:04:46.906947 25615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.903316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.2.208.190:43901)
0620 19:04:46.903460 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.903465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.906099 (+ 2634us) server_negotiation.cc:407] Connection header received
0620 19:04:46.906270 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.906273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.906322 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.906372 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.906835 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41358: BlockingRecv error: recv got EOF from 127.0.0.1:41358 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 19:04:46.908368 25766 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.903983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.2.208.188:35473)
0620 19:04:46.904095 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.904099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.907397 (+ 3298us) server_negotiation.cc:407] Connection header received
0620 19:04:46.907675 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.907679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.907728 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.907777 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.908253 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48934: BlockingRecv error: recv got EOF from 127.0.0.1:48934 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:04:46.912604 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.911125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.2.208.189:42881)
0620 19:04:46.911220 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.911224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.911715 (+ 491us) server_negotiation.cc:407] Connection header received
0620 19:04:46.911922 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.911925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.911999 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.912050 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.912498 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52658: BlockingRecv error: recv got EOF from 127.0.0.1:52658 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 19:04:46.917121 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.915752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.2.208.189:42881)
0620 19:04:46.915842 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.915846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.916345 (+ 499us) server_negotiation.cc:407] Connection header received
0620 19:04:46.916521 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.916525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.916568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.916615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.917037 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52674: BlockingRecv error: recv got EOF from 127.0.0.1:52674 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 19:04:46.922163 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.920817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.2.208.189:42881)
0620 19:04:46.920902 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.920905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.921379 (+ 474us) server_negotiation.cc:407] Connection header received
0620 19:04:46.921557 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.921560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.921603 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.921649 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.922080 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52676: BlockingRecv error: recv got EOF from 127.0.0.1:52676 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 19:04:46.929338 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.927948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.2.208.189:42881)
0620 19:04:46.928063 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.928068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.928544 (+ 476us) server_negotiation.cc:407] Connection header received
0620 19:04:46.928722 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.928726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.928773 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.928818 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.929256 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52690: BlockingRecv error: recv got EOF from 127.0.0.1:52690 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 19:04:46.940892 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.939292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.2.208.189:42881)
0620 19:04:46.939429 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.939435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.940010 (+ 575us) server_negotiation.cc:407] Connection header received
0620 19:04:46.940192 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.940197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.940247 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.940295 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.940791 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52706: BlockingRecv error: recv got EOF from 127.0.0.1:52706 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 19:04:46.961992 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.959199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.2.208.189:42881)
0620 19:04:46.959343 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.959348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.960924 (+ 1576us) server_negotiation.cc:407] Connection header received
0620 19:04:46.961159 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.961163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.961221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.961281 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.961874 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52716: BlockingRecv error: recv got EOF from 127.0.0.1:52716 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 19:04:46.997740 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:46.996443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.2.208.189:42881)
0620 19:04:46.996585 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:46.996590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:46.996829 (+ 239us) server_negotiation.cc:407] Connection header received
0620 19:04:46.996998 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:46.997007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:46.997055 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:46.997139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:46.997589 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52724: BlockingRecv error: recv got EOF from 127.0.0.1:52724 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:04:47.065281 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:47.063947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.2.208.189:42881)
0620 19:04:47.064208 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.064213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.064339 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:04:47.064539 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.064543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.064598 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.064691 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:47.065156 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52730: BlockingRecv error: recv got EOF from 127.0.0.1:52730 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
W20260620 19:04:47.196805 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:47.195598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.2.208.189:42881)
0620 19:04:47.195729 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.195734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.195867 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:04:47.196066 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.196069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.196117 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.196208 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:47.196665 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52744: BlockingRecv error: recv got EOF from 127.0.0.1:52744 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 19:04:47.456128 25692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:04:47.455115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.2.208.189:42881)
0620 19:04:47.455253 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.455257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.455332 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:04:47.455477 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.455480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.455526 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.455605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:04:47.455996 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52748: BlockingRecv error: recv got EOF from 127.0.0.1:52748 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:04:47.780509 25781 catalog_manager.cc:1259] Loaded cluster ID: 2949fe06eaec4f1688381165d98cbe4d
I20260620 19:04:47.780615 25781 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: loading cluster ID for follower catalog manager: success
I20260620 19:04:47.782531 25781 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: acquiring CA information for follower catalog manager: success
I20260620 19:04:47.783790 25781 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64f54b8044ee4143b37a27d526f18bdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:04:47.793196 25795 catalog_manager.cc:1259] Loaded cluster ID: 2949fe06eaec4f1688381165d98cbe4d
I20260620 19:04:47.793279 25795 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: loading cluster ID for follower catalog manager: success
I20260620 19:04:47.794847 25795 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: acquiring CA information for follower catalog manager: success
I20260620 19:04:47.796005 25795 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 933a0152e96c44398ad9b796887d5a61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:04:47.974725 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:47.970487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.2.208.189:42881)
0620 19:04:47.970779 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.970786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.970808 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:04:47.971002 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.971005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.971138 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.971256 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:47.971573 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.972125 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.973031 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.973268 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.973419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:47.973782 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:47.974280 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:47.974425 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:47.974484 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:04:47.974545 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260620 19:04:47.981864 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:47.977859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.2.208.188:35473)
0620 19:04:47.978133 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.978137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.978153 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:04:47.978296 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.978299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.978413 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.978543 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:47.978760 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.979431 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.980330 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.980560 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.980670 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:47.981006 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:47.981425 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:47.981580 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:47.981655 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:04:47.981708 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260620 19:04:47.989078 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:47.984788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.2.208.189:42881)
0620 19:04:47.985165 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.985169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.985182 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:04:47.985295 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.985298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.985351 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.985450 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:47.985740 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.986528 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.987776 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.988128 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.988243 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:47.988588 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:47.988662 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:47.988724 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:47.988901 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:04:47.988954 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 19:04:47.989600 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:47.984716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.2.208.190:43901)
0620 19:04:47.985468 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.985472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.985486 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:04:47.985615 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.985618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.985761 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.985880 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:47.986120 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.986955 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.988191 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.988426 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.988534 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:47.988859 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:47.989197 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:47.989286 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:47.989363 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:04:47.989429 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":65,"threads_started":1}
I20260620 19:04:47.990103 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:04:47.985374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.2.208.188:35473)
0620 19:04:47.985517 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:04:47.985523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:04:47.986171 (+ 648us) server_negotiation.cc:407] Connection header received
0620 19:04:47.986248 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:04:47.986251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:04:47.986296 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:04:47.986364 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:04:47.986638 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.987285 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.988536 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:04:47.988853 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:04:47.988925 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:04:47.989523 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:04:47.989593 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:04:47.989653 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:04:47.989960 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:04:47.990007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 19:04:47.992698 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25544
I20260620 19:04:47.999122 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25619
I20260620 19:04:48.005024 2882 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDDC2sW/build/debug/bin/kudu with pid 25693
2026-06-20T19:04:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3023 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6058 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279373 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST